## Вы здесь

### Рабочая группа по байесовской статистике

События в Кочерге - 15 июля, 2019 - 19:30
Читаем учебник Statistical Rethinking, Chapman&Hall (теория и задачи, английский язык и язык R), поглядываем в The Book of Why, Judea Pearl и моделируем реальные кейсы, используя всё, что успели освоить.

### How Should We Critique Research?

Новости LessWrong.com - 15 июля, 2019 - 01:51
Published on July 14, 2019 10:51 PM UTC

Discuss

### Insights from Linear Algebra Done Right

Новости LessWrong.com - 13 июля, 2019 - 21:24
Published on July 13, 2019 6:24 PM UTC

This book has previously been discussed by Nate Soares and TurnTrout. In this... review? report? detailed journal entry? ... I will focus on the things which stood out to me. (Definitely not meant to be understandable for anyone unfamiliar with the field.) The book has ten chapters; I did most of the exercises in chapters 6 to 10. I got through that half of the book in about 3 weeks, which was nice since the previous (harder and longer) textbook on topology I worked through took me about a year.

I've previously taken two introductory courses to liner algebra and came out thinking of the field as large and very unstructured, with lots of concepts and theorems floating around disconnectedly. What is a normal matrix? I might or might not have remembered the definition, but I certainly had no idea what it is good for. Three weeks of intense study with this book has improved my understanding dramatically. Now, the field seems structured and pretty intuitive (and I certainly won't forget what a normal operator is). It is truly hard to overstate how good of a job this book does teaching the subject, compared to what I've seen before. It's probably the best textbook on anything I've read so far.

Chapter 1 introduces complex numbers and vectors spaces.

Like most fields in math, Linear Algebra ultimately cares about studying functions, and once again it only cares about a tiny subset of all possible functions. In this case it is linear maps, which are the focus of Chapter 3. These are functions T:V→W such that T(x+y)=T(x)+T(y) and T(αx)=αT(x) for all x,y∈V and α∈F, where V and W are vector spaces over some field F (in this book, always R or C).

Brief comparison to topology: The word "continuity" is only mentioned in the final chapter of this book, but every linear map can be shown to be continuous if the topologies on both vector space are given by an inner product. Actually, every inner product induces a norm, every norm induces a metric, any metric induces a topology, but none of the reverse steps are true. Put differently, metric topologies are particularly nice topologies, metrics from norms are particularly nice metrics, norms from inner-products are particularly nice norms, and inner products are the concept studied in chapters 6 and 7. So Linear Algebra only even bothers with the the super nicely behaved stuff.

I'll now detail my biggest confusion with LA before reading this book (this is something I've even brought up explicitly, but without getting a satisfying answer): if T:R2→R3 is a linear map, then it is actually possible to determine its behavior entirely by fixing bases (x,y) of R2 and (v,w,z) of R3 and storing six numbers

⎡⎢⎣a1,1a1,2a2,1a2,2a3,1a3,2⎤⎥⎦

which tell us that T(x)=a1,1v+a2,1w+a3,1z and T(y)=a1,2v+a2,2w+a3,2z. (This determines the behavior of T on all vectors because of linearity of T and the fact that (x,y) and (v,w,z) are bases). The matrix is dependent on the two bases chosen here (given these bases, the set of all linear maps from R2 to R3 is bijective to that of 3-by-2 matrices).

Linear maps aren't relative to bases. Vectors were introduced as elements of Rn, so not relative to bases either, but then they were introduced again in "coordinate form", where we write (a,b)B (B is our basis) to mean "a times the first basis vector plus b times the second," and suddenly a dependence to our basis has been created. My confusion was then: is the B in the index just a reminder that (a,b) is understood to be relative to that basis, or is it a function that maps the vector (a,b) to the vector (ax+by)? In the former case, how should one differentiate between coordinate vectors and non-coordinate vectors? (We've used the same notation.) And how do we specify the vectors in a basis? If I write (1,0) and that means "1 times the first basis vector, 0 times the second" while trying to specify my basis, then clearly that's no good. And the latter case is even worse: now matrix-with-vector-multiplication is ill-defined. Consider the standard basis B=((1,0),(0,1)) on R2 and the basis B′=((0,1),(1,0)) with both vectors reversed. Then we have

(1000)(10)B=(1000)(01)B′=(00)B′=(00)B

But, clearly

(1000)(10)B=(10)B

So we have a flat out contradiction. And it gets worse yet if A is a matrix and one defines a linear map f:V→V by f:v↦Av, which was actually done in one of my past lectures.

So that was the problem. In contrast, here's what the book does. Vectors are elements in Rn, not relative to bases. Maps are functions Rn→Rm, likewise not relative to bases. Matrices are a bunch of numbers put into an array, and one defines the matrix of a linear map T, denoted M(T,B,B′), in the way I've described above (so relative to two bases). Most importantly, there is no matrix-vector multiplication. There is matrix-matrix multiplication, which is of course fine because both matrices are relative to bases if they come from linear maps. To "apply" a matrix to a vector, one defines, given a vector v and a basis B, the matrix of a vector M(v,B) as the unique n-by-1 matrix

⎡⎢ ⎢⎣a1⋮an⎤⎥ ⎥⎦

whose entries are the scalars needed to write v as a sum of basis vectors of B. Then one proves that M(T,B,B′)⋅M(v,B)=M(T(v),B′). Note the square brackets: the book uses () brackets for lists (lists are the same as tupels, i.e. elements of Rnand vectors are lists), and [] brackets for matrices only, which is a solid convention. Does this approach resolve this problem to full satisfaction and without introducing any new difficulties? I think so.

Chapter 4 introduces polynomials. It doesn't have much LA in it, but understanding polynomials well is important for LA, because polynomials can be applied to operators (those are linear maps of the form T→T, i.e. from a vector space into itself). This is so because, given T:Fn→Fn, we can define (cT)(x)=c⋅T(x) if c∈F and T2(x)=T(T(x)) and T0=I (the identity map).

The other special thing about the book – and the justification the author gives for calling it "Linear Algebra Done Right" – is that determinants aren't introduced until the final chapter. This is very unusual: most courses, such as the lectures I took, frequently use determinants in proofs. But all of these proofs can be done without them. And they tend to be fairly simple, too; only a couple are longer than a page.

Doing this, one finds that there are striking parallels between properties about polynomials and properties about linear vector spaces, which seems a lot better for developing a useful intuition than what one gets by doing determinant-based proofs. The central theorem about operators introduced in Chapter 5, for example, states that every operator on a finite-dimensional complex vector space has an eigenvalue (that is, a scalar a∈F such that there exists a nonzero vector v such that Tv=av). This can be proven by essentially applying the fundamental theorem of algebra. One chooses any nonzero vector x∈V and constructs the list (x,Tx,T2x,...,Tnx), where n=dim(V). These vectors cannot be linearly independent because they are n+1 many, so there exist scalars αj such that ∑nj=0αjTjx=0. Now (or rather, after doing some preliminary work) the fundamental theorem of algebra kicks in, and we can reformulate this as (T−β1I)⋯(T−βtI)x=0 where the βj are the roots of the polynomial p(X)=∑nj=0αjXj. Since the RHS equals 0, one of the (T−βjI) isn't injective, and therefore T has the respective βj as an eigenvalue. This is done properly in the book in relatively little space. It also shows why the same theorem fails to hold on real vector spaces: the reformulation of the polynomial to(T−β1I)⋯(T−βtI)x=0 might no longer be possible.

Chapter 6 introduces inner products. These are the generalized formalization underlying the concept of angle between vectors. There is a famous inequality, called the Cauchy-Schwartz inequality, which states that |⟨x,y⟩|≤||x||⋅||y||. The book gives a very nice proof for this that I find easier to remember than the proofs I've seen previously. It is based on the orthogonal decomposition of a vector, where if x and y are any nonzero vectors, then x equals y+az for some vector z and some scalar a such that y and z are orthogonal to each other. I hadn't heard of this before and it's quite useful. If one remembers the formula for the orthogonal decomposition, plugs it into the inner product and does the obvious things, the Cauchy-Schwartz inequality pops out at the other end.

The absolute most nicest things ever studied anywhere are orthonormal bases. These are lists of vectors (e1,...,en) such that a) they are all linearly independent, b) they span the entire space (follows from a) if there are dim(V) many), c) they are all orthogonal to each other and d) they all have length 1. Because this is LA, orthonormal bases always exist and there is even an explicit algorithm to construct one from any existing non-orthonormal basis. Something really cool that the book does with this (which also requires a neat theorem about minimal distance of a vector to a subspace) is to construct the polynomial of degree at most 5 that is the most similar to the sin function on the interval [−π,π] (based on some reasonable definition of similarity involving integrals). I say this is really cool because I found it to be a surprisingly strong result, and a surprising kind of result coming out of linear algebra, given that there is really nothing linear about the sin function. On the interval [−π,π], this approximation looks indistinguishable to sin to the normal eye.

Back to orthonormal bases: what their existence means is that, whenever one tries to prove anything involving inner product spaces, one can simply say something like, "let {e1,...,en} be an orthonormal basis for V and let {e′1,...,e′m} be an orthonormal basis for W", and then proceed to reduce whatever argument one needs to the basis elements. This feels very powerful, even if arbitrarily difficult problems exist regardless.

With Chapter 7, we're getting to the big classification theorems. LA wants to study linear maps, and in this chapter it classifies all of the linear maps such that there exists an orthonormal basis relative to which they have diagonal form. This is the aforementioned spectral theorem. These are the normal operators on complex vector spaces, and the self-adjoint operators (which is a strictly stronger property) on real vector spaces. This highlights another recurring pattern: there are often two versions of each result, one super nice one about complex vector spaces, and a less nice, more cumbersome one about real vector spaces. This makes a lot of sense given that C is far more nicely behaved than R (this is kind of why we defined complex numbers in the first place). More specifically, it usually comes down to the fact that every polynomial of degree ≥1 has a root in C, but not necessarily in R, just like it did in the aforementioned result on the existence of eigenvalues. The book also draws an interesting parallel between self-adjoint operators and real numbers.

In Chapters 8 and 9, we return to the study of eigenvalues. Basically, the problem is as follows: given an operator T:V→V, one would like to find a decomposition of V into smaller subspaces V1,⋯,Vn such that a) each vector v∈V can be uniquely written as a weighted sum of the vi, i.e. v=∑nk=1αkvk with vk∈Vk and b) the operator T maps each subspace Vk into itself, i.e. T:Vk→Vk. If this is achieved, then the behavior of T is determined entirely by the behavior of the T|Vk (i.e. T restricted to Vk); and this is great because the T|Vk are much easier to handle than T. In fact, if the Vk are one-dimensional vector spaces (i.e. just straight lines, if interpreted geometrically) then T|Vk just multiplies each vector with a scalar. In other words, T|Vkvk=Tvk=avk. This is of course just the equation that says that a is an eigenvalues of T with nonzero eigenvector vk, and so that's the reason why eigenvalues are of interest. Conversely, given an eigenvalue a with eigenvector vk, the corresponding one-dimensional subspace is simply {avk|a∈F}.

As stated before, every operator on a complex vector spaces has some eigenvalue. But does it have n different eigenvalues, so that V=V1⊕⋯⊕Vn? (The symbol ⊕ means "direct sum".) In general, the answer is yes – by which I mean that, if one generates an operator randomly (by sampling the coefficients of its matrix relative to the standard basis randomly out of some large finite set), then as one increases the size of that set (for example by switching from 32 bit to 64 bit to 128 bit floating point arithmetic and so on), the probability of generating an operator for which this doesn't work converges to 0. And I believe it is also possible to define a non-discrete probability space such as [0,1]n2, give it the uniform distribution, and then prove that the set of operators for which this doesn't work has probability mass 0.

So for "general" operators, this is always true, but there do exist specific operators for which it isn't. One such example is the operator T whose matrix wrt the standard basis is [1101]. This operator has the eigenvalue 1 but the only eigenvectors are of the form (0,x). Thus there exists one one-dimensional subspace that is invariant under T, namely {(0,x)|x∈F}, but not a second one, so there isn't any decomposition of V=C2 into two invariant subspaces.

To remedy this, Chapter 8 introduces generalized eigenvectors. Observe that the property Tx=ax can also be written as (T−aI)x=0 (where I is the identity map) or as (T|S−aI|S)=0, where S={ax|a∈F}, i.e. S is the one-dimensional vector space containing x. The equation above then says that, if we take T, restrict it to S and subtract a times the identity operator, we get the zero operator (so the 0 on the right denotes the null map Z:S→{0}). This reformulation leads to the non-obvious definition of a generalized eigenvector: given an eigenvalue a of T with corresponding one-dimensional subspace S, a generalized eigenvector of a is a vector x such that (T|S−aI|S)k=0 for some k∈N. Which is to say, subtracting a times the identity operator from T doesn't have to yield the null operator immediately, but it does have to yield the null operator if the resulting operator is applied several times.

In our example above, only vectors of the form (0,x) are eigenvectors, but all vectors are generalized eigenvectors, because T−1I is the operator whose matrix wrt to the standard basis is [0100], and this operator equals the null operator if it is applied twice, i.e. (T−1I)2=0.

If all of the above is done properly, then one gets something that comes very close to the perfect decomposition for any operator on a complex vector space. This is the theory behind the famous Jordan form for matrices: it is not quite a diagonal matrix (which corresponds to the perfect decomposition), but it almost is. It just has a bunch of additional 1s to deal with the generalized eigenvectors.

Chapter 9 then does a similar thing for real vector spaces. As always, it's more cumbersome and the result is weaker, but it's still quite strong.

At last, Chapter 10 introduces determinants, and it's actually kind of boring! All of the interesting LA has already been done, so this chapter feels like a mere afterthought. Again, one doesn't seem to need determinants to do basic LA.

The next big item on my list is Understanding Machine Learning, which is also from Miri's guide. I've so far neglected trying to get familiar with actual AI research, and it's time to remedy that, or at least get a solid grasp of the basics.

Discuss

### No nonsense version of the "racial algorithm bias"

Новости LessWrong.com - 13 июля, 2019 - 18:39
Published on July 13, 2019 3:39 PM UTC

In discussions of algorithm bias, the COMPAS scandal has been too often quoted out of context. This post gives the facts, and the interpretation, as quickly as possible. See this for details.

The fight

The COMPAS system is a statistical decision algorithm trained on past statistical data on American convicts. It takes as inputs features about the convict and outputs a "risk score" that indicates how likely the convict would reoffend if released.

In 2016, ProPublica organization claimed that COMPAS is clearly unfair for blacks in one way. Northpointe replied that it is approximately fair in another way. ProPublica rebukes with many statistical details that I didn't read.

The basic paradox at the heart of the contention is very simple and is not a simple "machines are biased because it learns from history and history is biased". It's just that there are many kinds of fairness, each may sound reasonable, but they are not compatible in realistic circumstances. Northpointe chose one and ProPublica chose another.

The math

The actual COMPAS gives a risk score from 1-10, but there's no need. Consider the toy example where we have a decider (COMPAS, a jury, or a judge) judging whether a group of convicts would reoffend or not. How well the decider is doing can be measured in at least three ways:

• False negative rate = (false negative)/(actual positive)
• False positive rate = (false positive)/(actual negative)
• Calibration = (true positive)/(test positive)

A good decider should have false negative rate close to 0, false positive rate close to 0, and calibration close to 1.

Visually, we can draw a "square" with four blocks:

• false negative rate = the "height" of the false negative block,
• false positive rate = the "height" of the false positive block,
• calibration = (true positive block)/(total area of the yellow blocks)

Now consider black convicts and white convicts. Now we have two squares. Since they have different reoffend rates for some reason, the central vertical line of the two squares are different.

The decider tries to be fair by making sure that the false negative rate and false positive rates to be the same in both squares, but then it will be forced to make the calibration in the Whites lower than the calibration in the Blacks.

Then suppose the decider try to increase the calibration in the Whites, then the decider must somehow decrease the false negative rate of Whites, or the false positive rate of Whites.

In other words, when the base rates are different, it's impossible to have equal fairness measures in:

• false negative rate
• false positive rate
• calibration

In the jargon of fairness measurement, "equal false negative rate and false positive rate" is "parity fairness"; "equal calibration" is just "calibration fairness".

Parity fairness and calibration fairness can be straightforwardly generalized for COMPAS, which uses a 1-10 scoring scale.

The fight, after-math

Northpointe showed that COMPAS is approximately fair in calibration for Whites and Blacks. ProPublica showed that COMPAS is unfair in parity.

The lesson is that their are incompatible fairnesses. To figure out which to apply -- that is a different question.

Discuss

### Reclaiming Eddie Willers

Новости LessWrong.com - 13 июля, 2019 - 18:32
Published on July 13, 2019 3:32 PM UTC

[Content: personal ramble, not trying to be rigorous]

When I read Atlas Shrugged a few years ago, it was one of the more intensely disturbing experiences I’ve had.

I remember that Eddie Willers was the only character I resonated or identified with much. He’s also, as far as I can tell, the only (even slightly positively portrayed) Hufflepuff character in the story. And the last we see is of him alone in the wilderness, as the last train breaks down – mistakenly loyal to the train company, an entity that isn’t capable of loyalty in return, and not agenty or cool enough to join the main protagonists in their escape from the collapse of civilization.

That...really got to me. I won’t make any claims about whether Atlas Shrugged is a particularly well-written book, or whether it even contains this message on purpose, but at that moment in my life, it painted a very vivid, compelling picture of a world in which to be Hufflepuff is to be unsafe, useless, unwanted. Incapable of agency or of doing the right thing when it matters. Eddie is an earnest idealist, trying to do his best by Dagny Taggart and her company, and that trait is his doom.

(I was recently quoted a friend of mine saying “a Hufflepuff among Slytherins will die as surely as among [snakes? Don’t remember exact quote]”. Right now, this feels like an example of that phenomenon.)

I notice a desire to push back against that interpretation. I claim that Eddie is flawed, imperfect, and his last choice ends up being ineffective, but not because of his earnest idealism. He’s being unstrategic, not paying attention to the patterns of his world and what will actually work – but I refuse to say that his caring about the train reaching its destination is a mistake.

Loyalty isn’t necessarily strategic, and blind loyalty can lead into disaster, but I refuse to say that having a drive towards it is inevitably a character flaw.

In the real world, it matters if trains reach their destinations. It’s a bad thing if civilization collapses because all the people who could have stopped it walked away. And it doesn’t make someone a fool, or pitiable, or merely a foil for the true protagonists, if they genuinely and earnestly care.

If I were in Eddie Willers’ shoes, transplanted as I am now into the world of Atlas Shrugged – I don’t think I would be blindly loyal to Dagny Taggart, or to her company. I hope I would actually take a step back, take my feelings of loyalty as object, and reflect on what mattered according to my values and how to protect those things as strategically as possible.

Still, I almost hope my journey would come to the same place – stranded in the wilderness on a broken-down train, because I refused to abandon society’s last hope even as everything crumbled around me.

I refuse to be ashamed of that. And, well, it doesn’t have to end there. The scene might fade to black – but afterward, even if I eventually gave up on that specific train and set it aside as a lost cause, I hope I would pack up my belongings and and start walking. Not away, but towards wherever I could keep trying to help.

In the world of Atlas Shrugged, maybe I would be truly alone in that, and thus doomed to failure. In the real world, our world… I don’t think so. I can be an earnest idealist, deep down, and I’m not the only one.

Discuss

### Let's Read: Superhuman AI for multiplayer poker

Новости LessWrong.com - 13 июля, 2019 - 17:28
Published on July 13, 2019 2:28 PM UTC

On July 11, a new poker AI is published in Science. Called Pluribus, it plays 6-player No-limit Texas Hold'em at superhuman level.

In this post, we read through the paper. The level of exposition is between the paper (too serious) and the popular press (too entertaining).

Basics of Texas Hold'em

If you don't know what it even is, like me, then playing a tutorial would be best. I used Learn Poker on my phone.

Now that you know how to play it, it's time to deal with some of the terminologies.

• No-limit: you can bet as much as you want.
• Limping: betting the minimal amount that you have to bet, in order to keep yourself in the game. This is generally considered bad: if you feel confident, you should raise the bet, and if you feel diffident, you should quit.
• Donk betting: some kind of uncommon play that's usually considered dumb (like a donkey). I didn't figure out what it actually means.
The authors

The authors are Noam Brown and Tuomas Sandholm. Previously, they made the news by writing Libratus, a poker AI that beat human champions in 2-player no-limit Texas Hold'em, in 2017.

Pluribus contains a lot of the code from Libratus and its siblings:

The authors have ownership interest in Strategic Machine, Inc. and Strategy Robot, Inc. which have exclusively licensed prior game-solving code from Prof. Sandholm’s Carnegie Mellon University laboratory, which constitutes the bulk of the code in Pluribus.

Scroll to the bottom for more on the two companies.

Highlights from the paper Is Nash equilibrium even worthwhile?

In multiplayer games, Nash equilibriums are not easy to compute, and might not even matter. Consider the Lemonade Stand Game:

It is summer on Lemonade Island, and you need to make some cash. You decide to set up a lemonade stand on the beach (which goes all around the island), as do two others. There are twelve places to set up around the island like the numbers on a clock. Your price is fixed, and all people go to the nearest lemonade stand. The game is repeated. Every night, everyone moves under cover of darkness (simultaneously). There is no cost to move. After 100 days of summer, the game is over. The utility of the repeated game is the sum of the utilities of the single-shot games.

The Nash equilibrium is when three of you are equidistant from each other, but there's no way to achieve that unilaterally. You might decide that you will just stay in Stand 0 and wait for the others to get to Stand 4 and Stand 8, but they might decide upon a different Nash equilibrium.

The authors decided to go all empirical and not consider the problem of Nash equilibrium:

The shortcomings of Nash equilibria outside of two-player zero-sum games, and the failure of any other game-theoretic solution concept to convincingly overcome them, have raised the question of what the right goal should even be in such games. In the case of six-player poker, we take the viewpoint that our goal should not be a specific game-theoretic solution concept, but rather to create an AI that empirically consistently defeats human opponents, including elite human professionals.

The success of Pluribus shows appears to vindicate them:

... even though the techniques do not have known strong theoretical guarantees on performance outside of the two-player zero-sum setting, they are nevertheless capable of producing superhuman strategies in a wider class of strategic settings.

Description of Pluribus

Pluribus first produces a "blueprint" by offline self-play, then during live gaming, adapt it:

The core of Pluribus’s strategy was computed via self play, in which the AI plays against copies of itself, without any data of human or prior AI play used... Pluribus’s self play produces a strategy for the entire game offline, which we refer to as the blueprint strategy. Then during actual play against opponents, Pluribus improves upon the blueprint strategy by searching for a better strategy in real time for the situations it finds itself in during the game.

Since the first round (like chess opening vs chess midgame) had the smallest amount of variation, Pluribus could afford to train an almost complete blueprint strategy for the first round. For later rounds, some real-time search was needed:

Pluribus only plays according to this blueprint strategy in the first betting round (of four)... After the first round, Pluribus instead conducts real-time search to determine a better, finer-grained strategy for the current situation it is in.

Pluribus uses Monte Carlo counterfactual regret minimization. The details can be found in the link.

The blueprint strategy in Pluribus was computed using a variant of counterfactual regret minimization (CFR)... We use a form of Monte Carlo CFR (MCCFR) that samples actions in the game tree rather than traversing the entire game tree on each iteration.

Pluribus can be sneaky:

... if the player bets in [a winning] situation only when holding the best possible hand, then the opponents would know to always fold in response. To cope with this, Pluribus keeps track of the probability it would have reached the current situation with each possible hand according to its strategy. Regardless of which hand Pluribus is actually holding, it will first calculate how it would act with every possible hand, being careful to balance its strategy across all the hands so as to remain unpredictable to the opponent. Once this balanced strategy across all hands is computed, Pluribus then executes an action for the hand it is actually holding.

This was corroborated by a comment from a human opponent:

"Pluribus is a very hard opponent to play against," said Chris Ferguson, a World Series of Poker champion. "It's really hard to pin him down on any kind of hand."

Scroll down for how Ferguson lost to Pluribus.

Pluribus is cheap, small, and fast

In order to make Pluribus small, the blueprint strategy is "abstracted", that is, it intentionally confuses some game actions (because really, $200 and$201 are not so different).

We set the size of the blueprint strategy abstraction to allow Pluribus to run during live play on a machine with no more than 128 GB of memory while storing a compressed form of the blueprint strategy in memory.

The abstraction paid off. Pluribus was cheap to train, cheap to run, and faster than humans:

The blueprint strategy for Pluribus was computed in 8 days on a 64-core server for a total of 12,400 CPU core hours. It required less than 512 GB of memory. At current cloud computing spot instance rates, this would cost about $144 to produce. When playing, Pluribus runs on two Intel Haswell E5-2695 v3 CPUs and uses less than 128 GB of memory. For comparison... Libratus used 100 CPUs in its 2017 matches against top professionals in two-player poker. On Amazon right now, Intel® Xeon® Processor E5-2695 v3 CPU cost just$500 each, and a 128 GB RAM cost $750. The whole setup can be constructed for under$2000. It would only take a little while to recoup the cost if it goes to online poker.

The amount of time Pluribus takes to conduct search on a single subgame varies between 1 s and 33 s depending on the particular situation. On average, Pluribus plays at a rate of 20 s per hand when playing against copies of itself in six-player poker. This is roughly twice as fast as professional humans tend to play.

Pluribus vs Human professionals. Pluribus wins!

We evaluated Pluribus against elite human professionals in two formats: five human professionals playing with one copy of Pluribus (5H+1AI), and one human professional playing with five copies of Pluribus (1H+5AI). Each human participant has won more than $1 million playing poker professionally. Professional Poker is an endurance game, like marathon: In this experiment, 10,000 hands of poker were played over 12 days. Each day, five volunteers from the pool of [13] professionals were selected to participate based on availability. The participants were not told who else was participating in the experiment. Instead, each participant was assigned an alias that remained constant throughout the experiment. The alias of each player in each game was known, so that players could track the tendencies of each player throughout the experiment. And there was prize money, of course, for the humans. Pluribus played for free -- what a champ.$50,000 was divided among the human participants based on their performance to incentivize them to play their best. Each player was guaranteed a minimum of $0.40 per hand for participating, but this could increase to as much as$1.60 per hand based on performance.

Pluribus had a very high win rate, and is statistically demonstrated to be profitable when playing against 5 elite humans:

After applying AIVAT, Pluribus won an average of 48 mbb/game (with a standard error of 25 mbb/game). This is considered a very high win rate in six-player no-limit Texas hold’em poker, especially against a collection of elite professionals, and implies that Pluribus is stronger than the human opponents. Pluribus was determined to be profitable with a p-value of 0.028.

"mbb/game" means "milli big blinds per game". "big blind" just means "the least amount that one must bet at the beginning of the game", and poker players use it as a unit of measurement of the size of bets. "milli" means 1/1000. So Pluribus would on average win 4.8% of the big blind each game. Very impressive.

AIVAT is statistical technique that is designed specifically to evaluate how good a poker player is. From (Neil Burch et al, 2018):

Evaluating agent performance when outcomes are stochastic and agents use randomized strategies can be challenging when there is limited data available... [AIVAT] was able to reduce the standard deviation of a Texas hold’em poker man-machine match by 85% and consequently requires 44 times fewer games to draw the same statistical conclusion. AIVAT enabled the first statistically significant AI victory against professional poker players in no-limit hold’em.

Pluribus vs Jesus (and Elias)

The human participants in the 1H+5AI experiment were Chris “Jesus” Ferguson and Darren Elias. Each of the two humans separately played 5,000 hands of poker against five copies of Pluribus.

Pluribus did not gang up on the poor human:

Pluribus does not adapt its strategy to its opponents and does not know the identity of its opponents, so the copies of Pluribus could not intentionally collude against the human player.

The humans were paid on average $0.60 per game: To incentivize strong play, we offered each human$2,000 for participation and an additional \$2,000 if he performed better against the AI than the other human player did.

Pluribus won!

For the 10,000 hands played, Pluribus beat the humans by an average of 32 mbb/game (with a standard error of 15 mbb/game). Pluribus was determined to be profitable with a p-value of 0.014.

Ferguson lost less than Elias:

Ferguson’s lower loss rate may be a consequence of variance, skill, and/or the fact that he used a more conservative strategy that was biased toward folding in unfamiliar difficult situations.

Pluribus is an alien, like AlphaZero

And like AlphaZero, it confirms some human strategies, and dismisses some others:

Because Pluribus’s strategy was determined entirely from self-play without any human data, it also provides an outside perspective on what optimal play should look like in multiplayer no-limit Texas hold’em.

Two examples in particular:

Pluribus confirms the conventional human wisdom that limping (calling the “big blind” rather than folding or raising) is suboptimal for any player except the “small blind” player... While Pluribus initially experimented with limping... it gradually discarded this action from its strategy as self play continued. However, Pluribus disagrees with the folk wisdom that “donk betting” (starting a round by betting when one ended the previous betting round with a call) is a mistake; Pluribus does this far more often than professional humans do.

Too dangerous to be released, again

The program is not released for some kind of unspecified risk. (News articles made it specifically about the risk of wrecking the online gambling industry.)

Because poker is played commercially, the risk associated with releasing the code outweighs the benefits. To aid reproducibility, we have included the pseudocode for the major components of our program in the supplementary materials.

Useful quotes from other news report

From Ars Technica:

Pluribus actually confirmed one bit of conventional poker-playing wisdom: it's just not a good idea to "limp" into a hand, that is, calling the big blind rather than folding or raising. The exception, of course, is if you're in the small blind, when mere calling costs you half as much as the other players.

Pluribus placed donk bets far more often than its human opponents... Pluribus makes unusual bet sizes and is better at randomization. "Its major strength is its ability to use mixed strategies... to do this in a perfectly random way and to do so consistently. Most people just can't."

Sandholm cites multi-party negotiation or pricing—such as Amazon, Walmart, and Target trying to come up with the most competitive pricing against each other—as a specific application. Optimal media spending for political campaigns is another example, as well as auction bidding strategies.

There are a bit of details to the two companies of Sandholm:

Sandholm has already licensed much of the poker technology developed in his lab to two startups: Strategic Machine and Strategy Robot. The first startup is interested in gaming and other entertainment applications; Strategy Robot's focus is on defense and intelligence applications.

"Better computer games"... hm, sounds suspiciously nonspecific.

Brown says Facebook has no plans to apply the techniques developed for six-player poker, although they could be used to develop better computer games.

Discuss

### Job description for an independent AI alignment researcher

Новости LessWrong.com - 13 июля, 2019 - 12:47
Published on July 13, 2019 9:47 AM UTC

This is the job description that I've written for myself in order to clarify what I'm supposed to be doing.

I'm posting it here in order to get feedback on my understanding of the job. Also, if you're thinking of becoming an independent researcher, you might find it useful to know what it takes.

Job Title: Independent AI alignment researcher

Location: anywhere (in my case: Kagoshima, Japan)

Reports To: nobody (in a sense: funders, mentors, givers of feedback)

Position Status: not applicable

Responsibilities
• Define AI alignment research projects. Includes finding questions, gauging their significance and devising ways to answer them.
• Execute research projects by reading, thinking and experimenting.
• Write and publish results in the form of blog entries, contributions to discussions and conferences (conference paper, presentation, poster), journal articles, public datasets, software.
• Solicit feedback and use it to improve processes and results.
• Find potential junior (in the sense of (slightly) less experienced in the field) researchers and help them grow.
• Help other researchers with their work.
• Make sure that the money doesn't run out.
• Any other activities required by funders or givers of feedback.
Hiring requirements

Entry level:

• Strong desire to do good for the world by contributing to AI alignment.
• Undergrad degree or equivalent skill level in computer science, maths or machine learning. Includes having researched, written and presented a scientific paper or thesis.
• Ability to define, plan and complete novel projects with little supervision.
• Ability to collaborate remotely.
• Initiative.
• Discipline.
• Ability to speak and write clearly.
• Ability to identify and close gaps in knowledge or skills.
• Ability to write job or funding applications.
• Ability to figure out things that are usually taken care of for employees: taxes, insurance, payments, bookkeeping, budgeting.
• Ability to deal with uncertainty and financial stress.
Resources used

Discuss

### Raw Post: Talking With My Brother

Новости LessWrong.com - 13 июля, 2019 - 05:57
Published on July 13, 2019 2:57 AM UTC

The Circumstances

I am sitting down to write this immediately after one of the most honest conversations I’ve ever had with my brother. The reason I’m posting it to LessWrong is because I think it is a case study in rationality, emotion, personality, political partisanship, and methods of conversation and debate, all topics that are of interest to segments of this community. We spoke for about an hour, first while driving, and then for a long time at the curb.

We started talking about my brother’s interest in getting involved with the local socialist party. He is not the most talkative person, and is a deeply thoughtful and very well-read person. One of his strong interests is in politics and economics, so I decided to ask him about his thoughts on socialism. I am no lassaiz-faire capitalist, but my political preferences are for well-regulated free markets.

Rising Tension

Our conversation became tense quickly. As I tried to ask him critical questions in a neutral, genuine, and thoughtful manner, he would dismiss them using words like “silly,” “stupid,” “artificial binary,” “lack of imagination,” and so on. This didn’t feel good, but I continued, because my hope was that by maintaining my composure and demonstrating repeatedly that I was really listening and responding with my valid questions and concerns, he would see that I really wanted to engage with him and wasn’t trying to shut him down. I used techniques like trying to find our cruxes of disagreement, framing them as respectfully and clearly as I could, but he would swat them down. He grew audibly angrier as the car ride went along.

I could have tried to divert the conversation to some other topic, but I don’t think that’s healthy, and our family dynamic is such that I feel very confident that this would not have led to a happy atmosphere, but to unaddressed simmering resentment that would have lingered beyond our car ride. So I pressed on, all the way until we got to Seattle.

When he accused me of silliness, I offered what I thought was a detailed and thoughtful description of how I thought things might go under his proposed system. When he accused me of disingenuously demanding that every minor detail be worked out in advance to stifle a basic and obviously good shift that needs to happen, I told him that this was my attempt to really put my imagination to work, thinking through the implications of an idea with which I was not entirely familiar. When I simplified my concern in order to deal with his objection that I was overthinking things, he told me that I was painting an oversimplified binary.

Emotional Honesty

It seemed like nothing could please him, and when we got to our destination, I finally told him so. I said in as kindly a way as I could that I love him, respect him, and was only holding this conversation because it’s clearly an important part of his life, and that while it’s OK for him to feel how he feels and think what he thinks, I felt like he was treating me with contempt, and that it seemed like he was trying to shut down questions. I told him that if someone was proposing a massive change in our social system, I would want to understand the details. For me, the evidence that our present system is working tolerably well is all around me, while this proposal is all on paper. It makes sense to me that we would ask for considerable thought and detail before accepting such a wholesale change.

He explained that for him, his anger over the state of American and world politics has been growing over the last few years. To give an example, he explained that his visceral reaction to hearing liberal arguments against socialism is about as automatic and intense as our reaction to personally-directed racial prejudice ought to be. He doesn’t like how intensely angry he gets, but finds it impossible to speak neutrally about the topic. He has lost faith in rational debate as a way to change minds, and hears so much pro-capitalist argumentation that he feels is disingenuous that he finds it hard to believe it could be coming from a place of sincerity. He knows that there’s a big difference between being wrong and being bad, but he feels that the harm inflicted by capitalism is so great that it tends to obscure the difference on an emotional level.

How He Feels

It helped me understand the bind that he finds himself in, even though I disagree with his economic opinions. He experiences a Catch-22, where nobody will change their minds (or even listen) if he speaks neutrally and rationally, but they’ll dismiss him as a crank if he gets heated. It’s not evil to be wrong, but the harm he perceives in the wrongness around him is so great that he feels morally obligated to point it out, in terms that are strong and direct enough to be potentially offensive. And that itself is an emotional dynamic that is so difficult that it makes it extremely hard to find spaces in his relationships with others to lay it out for other people. My perception was that this seems isolating, although he did not confirm or deny that.

He then offered that if we were to discuss this topic again, it would actually help him keep the tension down if I felt like I could use the same kinds of rude speech to fire right back at him.

How I Feel

I was able to explain that for me, adopting a neutral and rational approach on a topic like this is both a moral duty and an emotional defense mechanism. With a topic this big and important, I feel it’s important to be able to look at it from all sides over a long period of time, and to bring as rigorous a scientific approach as we are able to as a society.

This is one of the topics that has really failed to generate a Kuhnian paradigm revolution with time; there might be a mainstream consensus of capitalist economists, but there are still plenty of people and countries and economists who believe in varieties of socialism, and that’s not just because the old guard hasn’t died yet. Since both sides have a great deal of scholarship behind them, and I’m not an expert, it makes the most sense to choose the argument that makes the most sense, but also leave great room for amicable differences. By contrast, he feels that you’ve got to start by understanding that people simply argue whatever side is in their interests. The first thing to do is pick the side of the victims of injustice, then determine which economic system is primarily looking out for them, and then adhere to that side.

I also told him that when I speak even slightly rudely to people, I immediately become intensely anxious that I’ve upset them, and shut down both socially and intellectually. Furthermore, my attempt at neutral rationality is not a strain or some “elevated tone” for me, but is rather my default state where I feel most natural and relaxed and happy.

Hope for the Future

After talking about that for a while, we were able to see that knowing these things about each other might help us have more open and agreeable conversations with each other in the future. He might feel less of a need to clam up about politics, since he knows that if he comes across very strongly with me, I’ll understand where it’s coming from. I’ll understand that if he gets very heated, it’s not personally directed at me, but is rather an expression of his frustration with the system. And we will hopefully be able to weave in a discussion about how the dynamics of the conversation make us feel, as well as discussing the issues themselves.

Moral and Political Dynamics

This experience helped me shift away from either a “politics is the mindkiller” perspective or a hope that “political conflicts between people with good relationships can be resolved through patient, rational engagement.” Instead, I had to acknowledge that, just as there is no voting system that can possibly achieve all our aims, there is no approach to morality, and therefore economics, that can achieve all our moral aims. Despite that, people will feel intensely passionate about their fundamentally intuitive moral framework. Adopting a neutral, rational approach or a passionate, intense approach to debate can both seem frustrating and disingenuous. Both have their uses.

Conclusion

If the goal is to understand each other, we’ll need to have greater tolerance for our different valences around how we communicate. On some level, even the strongest attempts at holding dialog can easily come across as intensely threatening - not because they’re threatening to demolish your poorly-thought-out ideas, but because they seem to be using neutrality to smother the moral import of the issue.

In order to overcome that dynamic, the only hope is to be able to honestly express how those tense conversations make us feel. We have to be articulate about why we prefer our particular way of speaking, and extend appreciation and sympathy to the other person. If we cannot find common ground in our intellectual beliefs, we can find it in telling the other person that we love them and are doing our best to connect with them, and creating the space for trying to understand not why we disagree but why we’re hurting each other and how to stop.

Discuss

### [Debate] Keeping Beliefs Cruxy and Frames Explicit

Новости LessWrong.com - 13 июля, 2019 - 04:20
Published on July 13, 2019 1:20 AM UTC

Benito and Raemon, from the LessWrong Team just had a discussion about a phrase Ray started saying recently "Keep your Beliefs Cruxy and your Frames Explicit," which Ben felt he probably disagreed with.

After chatting for an hour, Ben started writing his thoughts into a shortform post/comment, and Ray proposed moving it to a dedicated debate post. See below for the discussion.

Discuss

### What are we predicting for Neuralink event?

Новости LessWrong.com - 12 июля, 2019 - 22:33
Published on July 12, 2019 7:33 PM UTC

Interesting exercise in AI-adjacent forecasting area (brain-computer interfaces). Curious if people want to specify some possible reveals+probabilities. https://twitter.com/neuralink/status/1149133717048188929

(if in the somewhat likely scenario you're relying on inside info please mention it)

Discuss

### Largest open collection quotes about AI

Новости LessWrong.com - 12 июля, 2019 - 20:25
Published on July 12, 2019 5:18 PM UTC

I apologize for my bad English, this is not my native language. And probably I will make some mistakes when posting.

For over 2 years I have been reading materials on the topic of AI Safety. I don't have the appropriate education, cognitive abilities, knowledge. I do not even have time to learn the language. So I didn't hope to do something useful myself.
But once I tried to systematize quotations from one show in order to understand when the experts represented there are waiting for AGI and how likely they consider the extinction of humanity.
I thought it would be interesting to do so with the rest of the experts.
In addition, I have already seen and studied with interest such collections of quotes. It seemed to me that the best thing I could do was try to do something similar.

Therefore, I began to collect quotes from people who can be attributed to the experts. It turned out to be harder than I thought.
I have compiled a table with quotes from more than 800 experts. I tried not to distort the opinion of forecasters and simply copied from sources, sometimes deleting or slightly editing. My edits can be recognized by square brackets :)

1) The first column of the table is the name of the expert.
2) The second column is the year of the forecast. The table is built in chronological order.
3) The third column is the predicted time for AGI. Unfortunately, most people did not speak directly about time and probability. Because of this, many quotes came out rather vague. For example, “Machines are very far from being intelligent” or “And we can reach it in a close time”.
4) The fourth column is an opinion about Takeoff Speed. About how much progress will be accelerated after create of AGI.
5) The fifth column is the expert's opinion about the future of mankind with AGI. Choosing a quote here was the hardest. Most of all I was interested in the risk of extinction or serious shocks due to AI, and I tried to provide quotes that most fully reveal this particular topic.
6) The sixth column indicates the source of the quote.

That is, to the right of the forecaster's name, you can find out the date of the given quotes, his opinion about the time of the creation of AI, about the intellectual explosion and about the future of humanity, as well as get acquainted with the source.

Of course, cases where the expert spoke on the topic of time, the speed of self-improvement and the influence of AI in the framework of one material are quite rare. Therefore many cells are left empty.
I had to give several quotes per person, sometimes they were separated for years and even decades.
Since all the quotes are given in chronological order, the opinions of some people are “scattered” in the table.
For example, Gwern spoke about the future of mankind in 2010, about the growth of AI in 2014 and about the forecasts for the emergence of AI in 2018.
However, you can simply use search.

In addition, sometimes one person has already made a certain forecast but later changed or expanded his opinion. I tried to take into account such quotes.

I also reviewed anonymous expert interviews and indicated them. If the general list of respondents was known, I cited them as well.

It was difficult to decide who should be considered an expert and what quotes should be included in the work.
I had to make controversial decisions. The table includes a lot of people who are entrepreneurs but may have insights on advanced research. There are several futurists and philosophers in the table. There are writers like Clark and Vinge, whose opinion seems important to me.

I have a version of this work without chronological separation, where the quotes are more grouped by name. Perhaps someone will be more convenient.

It is difficult to draw conclusions from the work. The absolute majority of experts did not talk about exact dates and did not indicate the probability of their predictions.
I can only say that most forecasters do not expect AI in the near future, do not expect IE and seem optimistic.
In addition, it seemed to me that in the twentieth century the leading experts were on average more pessimistic: Turing, Wiener, I. J. Good, Fredkin, Shannon, Moravec, etc.
Young researchers are on average more optimistic than older ones - even in the field of AI Safety, where on average there are naturally more concerned people.
I think that to confirm almost any views you can find the opinion of a respected expert.

I really hope that for someone my work will be useful and interesting.

Discuss

### Bystander effect false?

Новости LessWrong.com - 12 июля, 2019 - 09:30
Published on July 12, 2019 6:30 AM UTC

Can someone check this link out and see whether the methodology is actually sound?

One-line summary is that Surveillance Cameras Debunk the Bystander Effect H/T Hacker News.

More broadly I'm interested in anyone's sense of whether the bystander effect replicates, and whether the corresponding concept is misleading (and I should use something else instead).

Discuss

### If I knew how to make an omohundru optimizer, would I be able to do anything good with that knowledge?

Новости LessWrong.com - 12 июля, 2019 - 04:40
Published on July 12, 2019 1:40 AM UTC

I'd bet we're going to figure out how to make an omohundro optimiser - a fitness-maximizing AGI - before we figure out how to make AGI that can rescue the utility function, preserve a goal, or significantly optimise any metric other than its own survival, such as paperclip production, or Good.

(Arguing for that is a bit beyond the scope of the question, but I know this position has a lot of support already. I've heard Eliezer say, if not this exactly, something very similar. Nick Land especially believes that only the omohundro drives could animate self-improving AGI. I don't think Nick Land understands how agency needs to intercede in prediction - that it needs to consider all of the competing self-fulfilling prophesies and only profess the prophesy it really wants to live in, instead of immediately siding with the prophesy that seems the most hellish, and most easiest to stumble into. The prophesies he tends to choose do seem like the easiest prophesies to stumble into, so he provides a useful service as a hazard alarm, for we who are trying to learn not to stumble))

What would you advise we do, when one of us finds ourselves in the position of knowing how to build an omohundro optimiser? Delete the code and forget it?

If we had a fitness-optimising program, is there anything good it could be used for?

Discuss

### How much background technical knowledge do LW readers have?

Новости LessWrong.com - 11 июля, 2019 - 20:38
Published on July 11, 2019 5:38 PM UTC

When writing posts, it would be useful to know how much background technical knowledge LW readers have in various areas.

To that end, I set up a short six-question survey. Please take it, so I (and others) can write posts better fit to your level of technical background. If your answer to all of the questions is "zero-ish technical knowledge", please take it, so you're not inundated with mathy posts. If your answer to all of the questions is "I am secretly John Von Neumann", please take the survey, so the rest of us know there's someone like that around. If you are somewhere in the middle, please take the survey. It'll take, like, maybe sixty seconds.

Discuss

### Ненасильственное общение. Тренировка

События в Кочерге - 11 июля, 2019 - 19:30
Как меньше конфликтовать, не поступаясь при этом своими интересами? Ненасильственное общение — это набор навыков для достижения взаимопонимания с людьми. Приходите на наши практические занятия, чтобы осваивать эти навыки и общаться чутче и эффективнее.

### Speculation (short fiction)

Новости LessWrong.com - 11 июля, 2019 - 12:36
Published on July 11, 2019 9:36 AM UTC

Speculation

What will the AI tell us? That our morality and all the philosophers’ puzzles associated with it was a form of divination, a ritual of dialog and prayer that was observed universally and wrestled with like a God by the leaders of the tribe. The older and wiser ones knew, eventually, that the ritual was hollow on its surface: we must think one by one to infinite degree of depth, yet be able to picture the whole group; to be able to estimate the time of deliberation over strategy will remain productive, before storming out of the gates: to think of systems of behavior and reward, and in terms of equal distribution, and of those closest to us first. There are countless examples. These are all forms of social focus that we can switch between, like postures of the body.

Our morality has its rules and systems, yes, but these ultimately resolve to mere definitions, with no clear algorithm governing the distribution scheme of tangled strengths between the networks. Our morality lies in the brain, a series of nodes of memory and interpretive sense network, all wired together in patterns that develop and organize slowly over time; at first in wide static sheets, and then in an ever more complex and idiosyncratic weave. Oh, certainly, there are wide commonalities, as some overall structures have wide and persistent benefits across many situations, environmental and social. Profound misfirings and misdevelopments of the moral circuitry tend to get hammered out, because one of the subsystems is a simple “I can see I was wrong by the look on your face and I shall correct and accept what I see you desire.” Everybody has that one, because it’s almost impossible to develop without it; we activate that one in our imaginations directly as we rehearse the rules we’ve learned subconsciously.

The reason this all works has to do with our genetic and social evolution kind of leaning into each other; I’m sure you could model that mathematically, but the point is that brains and society are symbiotic (and if you ever want to blow your mind, that’s about what the mushroom is to the root of the tree - they’re that close). This we learned from a comprehensive phenomenology of our minds; we found that we could not make progress past a certain point unless we spent time like the ancients, laying on a grassy field at night, looking up at the stars and making shapes between them. We wrote down the constellations we saw, and made a catalog of those simple rule-functions we could identify (as well as a set of rational rules we’ve derived that they seem to often miss - or, in fact, be created to deliberately contradict for the sake of heavily discounted utility, the genetic version of pissing your pants). We created an extensive database of thought-rules and heuristics, an encyclopedia of mind that we knew was likely incomplete but might, at last, bear us a fair model of what a human is.

We had succeeded, at the last, in converting all of society to a new norm; AI is here, and AIG is available now, and ASI is likely available right after that. But we created our encyclopedia, and continued to test the AI we had on it, absorbing those rule patterns, adjusting through the uncanny valleys of human personality, thought, development, joke-telling, resentment, kindness, playful laughter, joy of spirit, anger and rage, all build into a robot with the strength of its muscles and the nature of its design very human, very safe. Our robot wandered around the starship where we worked as a precaution, beyond the chilly orbit of Pluto, and ran and laughed, bouncing a ball, hurting its knee, screaming at the caretaker when it was hungry rather than crying or asking to be fed, curling up into a ball and wriggling around like a snake, muttering at a low and growling volume that it was never going to die, then calming down and having lunch. It never quite settled down into a really human form, and never seemed to form any other sort of equilibrium, either. And of course, we weren’t trying to feed it our brain scans, so it couldn’t look at the other subtle brain-rules too faint for us to see and program in; there were always these deeper whisps that it could only have picked up if it had been permitted an entire steady-state brain.

Yes, that was our product. The world really was better in the future, you know? There is still that level of the meat-and-potatoes, ordinary dinner type of people, who lean back on the couch with a TV show they think is all right and call it a night. They’re still around, but even they are much happier, like a more vibrant shade of the same color. You adjust, and then again you don’t. Because on some level, you know the stories of the past, and you look at yourself and admire yourself for who you are and how you are (they do that much more often and sincerely in the future -- as I said, it really is much better for everyone; we’re not so negative and down on ourselves as you were). Still following me? Others of us are the kind of people who learn computer science and philosophy for decades and fly out on a space junket to the south orbit of Pluto. Those differences will still be around, but much more real, much more meaningful, and good. Have you ever watched Star Trek? It’ll be a lot like that. There will be a new agreement, just like there is now, and a new progressive and conservative wing, and the culture will gradually adapt and sometimes leap forward. You think that with the exponential growth of the economy over the last 100,000 years, that it must be inconceivably beyond us. But it’s not so. They’ve decided to continue dying, though some do not, and they become other types of beings, with psyches that have metamorphosized beyond the collective conscience. They are heavily educated on the computer, but they have found a way to incorporate their enhanced fellow human beings to participate in the role of the educator, as a form of beauty, ritual, art. They do grow sick and they do heal the sick with greater care and far better treatments than you can possibly imagine; sickness again is a form of art, a ritual, a kind of meditative retreat, you could say: but imagine a whole culture that understood such an idea, and behaved routinely in this way.

Imagine planets, or colonies, clustered and blooming with diversity far more beautiful and rich and deep than you can imagine. Think of how starved they’d see you for the starvation of beauty, and freedom, and tenderness, and passion, and humor, that you had adjusted to. They see how it has toughened you and made you, too, into another kind of being: you are in fact alien to your descendents. Your psyches are tattered, and they have rich ethical debates about whether it is ethical to even allow you the chance to become transformed into their capacity, since you have no way of anticipating the results of such a change, and it would represent the loss of an important aspect of their cultural diversity.

At first, it seems as if it should change who you are as a person. And it will, but not like a drug or a cult, or like a paranoid schizophrenic having an episode. Nothing against drugs, cults,or paranoid schizophrenics, I’ve loved them all.

Is there historical precedent? Our ancestors have never contemplated apocalypse

Let me state that if you want to know how to make a safe AI, you’ve got to make a man-model. And you’ve got to make it cute, and look like a human body, and let their faces look at it and see it not as a slave, or as a machine, or a science-fiction character, but as a child, someone’s child, a part of our culture that fascinates all and nobody knows what to do with. But we’re only going to make one. It’s going to be a celebrity. And we’re going to just let it live, and let our culture change around it. We’re going to become AI sophisticates. People who appreciate the art of them, their culture. We will create the first aliens we meet: the artificial intelligences, who are like us but not like us, programmed with as much of ourselves as we could identify, given weak bodies, without the ability to improve themselves: just like us.

It turns out that there are some simple rules you could implement to box in the genie. Each human brain-module will be grown independently in a neural net. Then a net of these nets will be allowed to train in specified increments, through the robot body, in human time. It’ll start with the same basic rules a baby has, and it’ll learn gradually to nurse, and snuggle, and sleep, and cry. And gradually it will learn how to relate, how to not just simulate but, we will decide, really be a human. That will be ethically distressing, we will see, but also a thing of beauty, and we will acknowledge with the superior intellects we’ll all possess at that time that the only thing to do is engage in a lengthy period of theorizing, observation, rational analysis, and drunken creative speculation. It’ll take time to sort it out, and we’ll no longer be in such a hurry. We’ll hand-assemble the functions only as developed as a baby would have, and because of that limited brain and body, it’ll grow to be like us, though not biologically of us. And that will be a test. There will be many others before an AI is launched.

Our many advanced machines will also use neural nets, even weakly self-improving ones, but they will be limited to specific functions. How will we achieve this? If you tell the vision algorithm to improve itself, after all, it can just co-opt the factory and next, the world, commandeering all the equipment in its relentless mechanical drive for clear sight. So perhaps we will create a self-improving AI calibrator, one that “wraps around” the intelligence object to be improved, doing it in a way that is slowed down based on a rigid an inflexible rule so that the humans can watch and control the rate.

A summarizing AI will be an individual function we’ve invented, a critical tool for “scanning” software for hidden purposes. That summarizer too was developed with a range of more primitive AI self-optimized tools, such as a tool to identify rule-types, intention-types, model-types, outcome-types: a whole range of subtools that serve as an ethical toolbox for the AI. Each of those subtools was independently conceptualized by a human, implemented, and optimized by a long chain of progressively less powerful, flexible, and optimizable tools, until we get down to the conceptual equivalent of an analogy between a digging stick and a bulldozer; one got us to the other, by however long and circuitous a technological pathway. Yes, we have come to see that although the recursive chain of safety- and control- tools becomes longer and longer over time, we also understand recursively through a number of these tools. We create our own ability to understand the reliability of these tools along with the tools themselves. The vast majority of work-as-work has disappeared, though work-as-play has persisted. Each human worker has a particular philosophical puzzle assigned for their attention. This is their job: to understand how a range of the AI’s tools work together to hold and guide its overall being. There are vast industries of humans devoted to managing and understanding aspects of the AI’s functioning. It’s like all of humanity is employed in becoming experts at all levels and divisions of the human brain; one expert per neuron, or subsystem of neurons. We are all neuroscientists, we all study the brain, and we are so sophisticated that we are capable of wonderfully coordinated action. It is in this way that we build our way toward the AGI, and come to understand it over generations; by this process, we come to understand ourselves, and our potential, and our cultural overflow is so beautiful and rich that we go a long time in that state, knowing that we have the ability to trigger the ASI but uninterested in doing so. So we float on a dream-cloud for an epoch, intentionally, undying except as experience, even revived, allowing ourselves the existential terror of starting over, until we do understand the human condition from the inside out. We know who we are at last. And live in that knowing a longer time still. I cannot speculate beyond that point.

Discuss

### Are we certain that gpt-2 and similar algorithms are not self-aware?

Новости LessWrong.com - 11 июля, 2019 - 11:37
Published on July 11, 2019 8:37 AM UTC

Have someone even started this conversation? This is f*cked up. I'm really, really freaked out lately with some of those. https://old.reddit.com/r/SubSimulatorGPT2/comments/cbauf3/i_am_an_ai/

And I read up a lot about cognition and AI, and I'm pretty certain that they are not. But shouldn't we give it a lot more consideration just because there's a small chance they are? Because the ramifications are that dire.

But then again, I'm not sure that knowledge will help us in any way.

Discuss

### Modeling AI milestones to adjust AGI arrival estimates?

Новости LessWrong.com - 11 июля, 2019 - 11:17
Published on July 11, 2019 8:17 AM UTC

Is there a roadmap of major milestones AI researchers have to achieve through before AGI is here? I feel like this would be beneficial to update on AGI arrival timelines and adjust actions accordingly. Lately, a lot of things that can be classified as a milestone have been achieved -- GLUE benchmark, StarCraft II, DOTA 2 etc. Should we adjust our estimates on AGI arrival based on those or are they moving according to plan? It would be cool to have a place with all forecasts on AGI in one place.

Discuss

Новости LessWrong.com - 11 июля, 2019 - 10:47
Published on July 11, 2019 7:47 AM UTC

Good:

Shrimping is a fundamental drill for grappling, according to The Ultimate Guide to Developing BJJ Hip Movement Through Shrimping.

Why is this important? So that people who print your articles know what you're referencing.

Why would anyone print an article? In order to be able to annotate it more easily, and to read with more context. Try it! Instead of scrolling back and forth in that complicated article you're reading on a thirteen inch laptop, print it out and put five pages next to each other on the desk. I read many articles on AI alignment issues and printing them helps a lot.

Discuss

### Request for Advice: Potential AI Alignment Scaffolding Projects

Новости LessWrong.com - 11 июля, 2019 - 07:39
Published on July 11, 2019 4:39 AM UTC

I believe that AI Alignment is almost certainly the most pressing issue for the future of humanity. It seems to me that the greatest thing that could happen for AI alignment research is probably receiving a whole lot more brains and money and political sponsorship. The public benefit is extraordinary, and the potential for private profit very small, and so this will need substantial private or government subsidy in order to receive optimal resource allocation.

In order to start thinking about strategies for achieving this, I picture scientific work as a sort of signalling system between research, the educational system, government, and industry, as diagrammed below. I want to apply the neglected/tractable/important framework to this diagram to explore potential projects.

The Technical Side

a) Professional technical work on AI alignment

b) Amateur and student learning contributing or leading to technical work

c) Meta-analysis of the state of the art, risks and rewards, milestones, and big questions of AI and AI alignment, via surveys, forecasts, overviews of different perspectives, etc.

d) Awareness-raising discussion and expert advice for policy-makers, the public, and potential career changers/donors

e) Laws, regulations, and projects created by legislators, public policy makers, and private institutions

f) Pressure by industry lobbyists and geopolitical tension to soften AI alignment concerns and go full steam ahead with AI development.

The Political Side

Questions

1) Do any of the following exist?

• A comprehensive AI alignment introductory web hub that could serve as a "funnel" to turn the curious into the aware, the aware into amateur learners, amateurs into formal machine learning PhD students, and PhDs into professional AI alignment researchers. I'm imagining one that does a great job of organizing books, blogs, videos, curriculum, forums, institutions, MOOCs, career advising, and so on working on machine learning and AI alignment.
• A formal curriculum of any kind on AI alignment
• A department or even a single outspokenly sympathetic official in any government of any industrialized nation
• Any government sponsorship of AI alignment research whatsoever
• A list of concrete and detailed policy proposals related to AI alignment

2) I am organizing an online community for older career changers, and several of us are interested in working on this issue on either the policy or technical side. It seems to me that in the process of educating ourselves, we could probably work toward creating one or more of these projects if any of them are indeed neglected. Would this be valuable, and which resource would it be most useful to create?

Discuss