Вы здесь

Сборщик RSS-лент

Paper-Reading for Gears

Новости LessWrong.com - 5 декабря, 2019 - 00:02
Published on December 4, 2019 9:02 PM UTC

Lesswrong has a fair bit of advice on how to evaluate the claims made in scientific papers. Most of this advice seems to focus on a single-shot use case - e.g. a paper claims that taking hydroxyhypotheticol reduces the risk of malignant examplitis, and we want to know how much confidence to put on the claim. It’s very black-box-y: there’s a claim that if you put X (hydroxyhypotheticol) into the black box (a human/mouse) then Y (reduced malignant examplitis) will come out. Most of the advice I see on evaluating such claims is focused around statistics, incentives, and replication - good general-purpose epistemic tools which can be applied to black-box questions.

But for me, this black-box-y use case doesn’t really reflect what I’m usually looking for when I read scientific papers.

My goal is usually not to evaluate a single black-box claim in isolation, but rather to build a gears-level model of the system in question. I care about whether hydroxyhypotheticol reduces malignant examplitis only to the extent that it might tell me something about the internal workings of the system. I’m not here to get a quick win by noticing an underutilized dietary supplement; I’m here for the long game, and that means making the investment to understand the system.

With that in mind, this post contains a handful of thoughts on building gears-level models from papers. Of course, general-purpose epistemic tools (statistics, incentives, etc) are still relevant - a study which is simply wrong is unlikely to be much use for anything. So the thoughts and advice below all assume general-purpose epistemic hygiene as a baseline - they are things which seem more/less important when building gears-level models, relative to their importance for black-box claims.

I’m also curious to hear other peoples’ thoughts/advice on paper reading specifically to build gears-level models.

Get Away From the Goal

Ultimately, we want a magic bullet to cure examplitis. But the closer a paper is to that goal, the stronger publication bias and other memetic distortions will be. A flashy, exciting result picked up by journalists will get a lot more eyeballs than a failed replication attempt.

But what about a study examining the details of the interaction between FOXO, SIRT6, and WNT-family signalling molecules? That paper will not ever make the news circuit - laypeople have no idea what those molecules are or why they’re interesting. There isn’t really a “negative result” in that kind of study - there’s just an open question: “do these things interact, and how?”. Any result is interesting and likely to be published, even though you won’t hear about it on CNN.

In general, as we move more toward boring internal gear details that the outside world doesn’t really care about, we don’t need to worry as much about incentives - or at least not the same kinds of incentives.

Zombie Theories

Few people want to start a fight with others in their field, even when those others are wrong. There is little incentive to falsify the theory of somebody who may review your future papers or show up to your talk at a conference. It’s much easier to say “examplitis is a complex multifactorial disease and all these different lines of research are valuable and important, kumbayah”.

The result is zombie theories: theories which are pretty obviously false if you spend an hour looking at the available evidence, but which are still repeated in background sections and review articles.

One particularly egregious example I’ve seen is the idea that a shift in the collagen:elastin ratio is (at least partially) responsible for the increased stiffness of blood vessels in old age. You can find this theory in review articles and even textbooks. It’s a nice theory: new elastin is not produced in adult vasculature, and collagen is much stiffer, so over time we’d expect the elastin to break down and collagen to bear more stress, increasing overall stiffness. But if we go look for studies which directly measure the collagen:elastin ratio in the blood vessels… we mostly find no significant change with age (rat, human, rat); one study even finds more elastin relative to collagen in older humans.

Ignore the Labels on the Box

Scientists say lots of things which are misleading, easily confused, or aren’t actually supported by their experiments . That doesn’t mean the experiment is useless, it just means we should ignore the mouth-motions and look at what the experiment and results actually were. As an added bonus, this also helps prevent misinterpreting what the paper authors meant.

An example: many authors assert that both (1) atherosclerosis is a universal marker of old age among humans and most other mammals, and (2) atherosclerosis is practically absent among most third-world populations. What are we to make of this? Ignore the mouth motions, look for data. In this case, it looks like atherosclerosis does universally grow very rapidly with age in all populations examined, but still has much lower overall levels among third-world populations after controlling for age - e.g. ~⅓ as prevalent in most age brackets in 1950’s India compared to Boston.

Read Many Dissimilar Papers: Breadth > Depth

For replication, you want papers which are as similar as possible, and establishing very high statistical significance matters. For gears-level models, you want papers which do very different things, but impinge on the same gears. You want to test a whole model rather than a particular claim, so finding qualitatively different tests is more important than establishing very high statistical significance. (You still need enough statistics to make sure any particular result isn’t just noise, but high confidence will ultimately be established by incrementally updating on many different kinds of studies.)

For example, suppose I’m interested in the role of thymic involution as a cause of cancer. The thymus is an organ which teaches new adaptive immune cells (T-cells) to distinguish our own bodies from invaders, and it shrinks (“involutes”) as we age.

Rather than just looking for thymus-cancer studies directly, I move away from the goal and look for general information on the gears of thymic involution. Eventually I find that castration of aged mice (18-24 mo) leads to complete restoration of the thymus in about 2 weeks. The entire organ completely regrows, and the T-cells return to the parameters seen in young mice. (Replicated here.) Obvious next question: does castration reduce cancer? It’s used as a treatment for e.g. prostate cancer, but that’s (supposedly) a different mechanism. Looking for more general results turns up this century-old study, which finds that castration prevents age-related cancer in mice - and quite dramatically so. Castrated old mice’ rate of resistance to an implanted tumor was ~50%, vs ~5% for controls. (This study finds a similar result in rabbits.) Even more interesting: castration did not change the rate of tumor resistance in young mice - exactly what the thymus-mediation theory would predict.

This should not, by itself, lead to very high confidence about the castration -> thymus -> T-cell -> cancer model. We need more qualitatively different studies (especially in humans), and we need at least a couple studies looking directly at the thymus -> cancer link. But if we find a bunch of different results, each with about this level of support for the theory, covering interventions on each of the relevant variables, then we should have reasonable confidence in the model. It’s not about finding a single paper which proves the theory for all time; it’s about building up Bayesian evidence from many qualitatively different studies.

Mediation is Everything

Everything is correlated with everything else; any intervention changes everything.

That said, very few things are directly connected; the main value is finding variables which mediate causal influence. For instance, maybe hydroxyhypotheticol usually reduces malignant examplitis, but most of the effect goes away if we hold hypometabolicol levels constant. That’s a powerful finding: it establishes that hypometabolicol is one of the internal gears between hydroxyhypotheticol and examplitis.

If I had to pick the single most important guideline for building gears-level models from papers, this would be it: mediation is the main thing we’re looking for.


On decision-prediction fixed points

Новости LessWrong.com - 5 декабря, 2019 - 00:02
Published on December 4, 2019 8:49 PM UTC

It seems like for embedded (reflexive, Löbian, etc) LDT agents, there ought to be a fixed point thing between decision and prediction.

Indeed, embedded agents can predict things about their own actions; but by modeling themselves sufficiently well, this should be (in the limit) equivalent to making a decision, as they will be modeling their own thoughts. Conversely, once you have decided, if you do not suffer from akrasia, then you have accurately predicted your next action. (aside: this is the source of the illusion of free will.)

This is related to the class of "metaphysical truths": truths of the form ☐P → P. Whenever an embedded agent believes one of those, then it must (by Löb's theorem) eventually believe P. But there are lots of such truths (perhaps each different religion offers a different set of metaphysical truths), which might then lead to spurious, or even contradictory beliefs!

The key word was "eventually", assuming LDT agents are logical inductors of some kind; in the meantime, the agent may choose its beliefs. Isn't this weird? Beliefs shouldn't be arbitrary!

But you can imagine, as an (imperfect) example, the paradox of self-confidence: if you think you are competent, then you could believe in your ability to self-improve, which will encourage your to foster your own competence; on the other hand thinking that you are incompetent may lead to not believing in your self-improvement ability, leading to a downward spiral.

Each one of these are decision-belief fixed points. Each are, in way (causally?), both true and rational.

I feel like LDT will end up being a reflexive fixed point of this sort (reminiscent of the logical induction fixed point), with the catch that there are many such fixed points. The true decision an LDT agent must make is then choosing the most effective of these fixed points.

(I'm not entirely convined of this yet since I still have no idea what logical counterfactuals will look like)

The moral of the story for us humans is that:

  • akrasia should not exist, not if you can predict yourself well enough;
  • sometimes beliefs are arbitrary. choose the most productive ones, you'll end up believing them all anyway.


What additional features would you like on LessWrong?

Новости LessWrong.com - 4 декабря, 2019 - 22:41
Published on December 4, 2019 7:41 PM UTC

I'm not on the LessWrong team; I'm just curious, and might want to answer that question myself ^_^


[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations

Новости LessWrong.com - 4 декабря, 2019 - 21:10
Published on December 4, 2019 6:10 PM UTC

[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations View this email in your browser Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version here (may not be up yet).


Self-training with Noisy Student improves ImageNet classification (Qizhe Xie et al) (summarized by Dan H): Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.

Dan H's opinion: Some in the safety community have speculated that robustness to data shift (sometimes called "transfer learning" in the safety community) cannot be resolved only by leveraging more GPUs and more data. Also, it is argued that the difficulty in attaining data shift robustness suggests longer timelines. Both this paper and Robustness properties of Facebook's ResNeXt WSL models analyze the robustness of models trained on over 100 million to 1 billion images, rather than only training on ImageNet-1K's ~1 million images. Both papers show that data shift robustness greatly improves with more data, so data shift robustness appears more tractable with deep learning. These papers evaluate robustness using benchmarks collaborators and I created; they use ImageNet-AImageNet-C, and ImageNet-P to show that performance tremendously improves by simply training on more data. See Figure 2 of the Noisy Student paper for a summary of these three benchmarks. Both the Noisy Student and Facebook ResNeXt papers have problems. For example, the Noisy Student paper trains with a few expressly forbidden data augmentations which overlap with the ImageNet-C test set, so performance is somewhat inflated. Meanwhile, the Facebook ResNeXt paper shows that more data does not help on ImageNet-A, but this is because they computed the numbers incorrectly; I personally verified Facebook's ResNeXts and more data brings the ImageNet-A accuracy up to 60%, though this is still far below the 95%+ ceiling. Since adversarial robustness can transfer to other tasks, I would be surprised if robustness from these models could not transfer. These results suggest data shift robustness can be attained within the current paradigm, and that attaining image classifier robustness will not require a long timeline.

Safety Gym (Alex Ray, Joshua Achiam et al) (summarized by Flo): Safety gym contains a set of tasks with varying difficulty and complexity focused on safe exploration. In the tasks, one of three simulated robots has to move to a series of goals, push buttons or move a box to a target location, while avoiding costs incurred by hitting randomized obstacles. This is formalized as a constrained reinforcement learning problem: in addition to maximizing the received reward, agents also have to respect constraints on a safety cost function. For example, we would like self-driving cars to learn how to navigate from A to B as quickly as possible while respecting traffic regulations and safety standards. While this could in principle be solved by adding the safety cost as a penalty to the reward, constrained RL gets around the need to correctly quantify tradeoffs between safety and performance.

Measures of safety are expected to become important criteria for evaluating algorithms' performance and the paper provides first benchmarks. Constrained policy optimization, a trust-region algorithm that tries to prevent updates from breaking the constraint on the cost is compared to new lagrangian versions of TRPO/PPO that try to maximize the reward, minus an adaptive factor times the cost above the threshold. Interestingly, the lagrangian methods incur a lot less safety cost during training than CPO and satisfy constraints more reliably at evaluation. This comes at the cost of reduced reward. For some of the tasks, none of the tested algorithms is able to gain nontrivial rewards while also satisfying the constraints.

Lastly, the authors propose to use safety gym for investigating methods for learning cost functions from human inputs, which is important since misspecified costs could fail to prevent unsafe behaviour, and for transfer learning of constrained behaviour, which could help to deal with distributional shifts more safely.

Flo's opinion: I am quite excited about safety gym. I expect that the crisp formalization, as well as the availability of benchmarks and ready-made environments, combined with OpenAI's prestige, will facilitate broader engagement of the ML community with this branch of safe exploration. As pointed out in the paper, switching from standard to constrained RL could merely shift the burden of correct specification from the reward to the cost and it is not obvious whether that helps with alignment. Still, I am somewhat optimistic because it seems like humans often think in terms of constrained and fuzzy optimization problems rather than specific tradeoffs and constrained RL might capture our intuitions better than pure reward maximization. Lastly, I am curious whether an increased focus on constrained RL will provide us with more concrete examples of "nearest unblocked strategy" failures, as the rising popularity of RL arguably did with more general examples of specification gaming.

Rohin's opinion: Note that at initialization, the policy doesn't "know" about the constraints, and so it must violate constraints during exploration in order to figure out what the constraints even are. As a result, in this framework we could never get down to zero violations. A zero-violations guarantee would require some other source of information, typically some sort of overseer (see delegative RL (AN #57), avoiding catastrophes via human intervention, and shielding).

It's unclear to me how much this matters for long-term safety, though: usually I'm worried about an AI system that is plotting against us (because it has different goals than we do), as opposed to one that doesn't know what we don't want it to do.

Read more: Github repo

Technical AI alignment   Problems

Classifying specification problems as variants of Goodhart's Law (Victoria Krakovna et al) (summarized by Rohin): This post argues that the specification problems from the SRA framework (AN #26) are analogous to the Goodhart taxonomy. Suppose there is some ideal specification. The first step is to choose a model class that can represent the specification, e.g. Python programs at most 1000 characters long. If the true best specification within the model class (called the model specification) differs from the ideal specification, then we will overfit to that specification, selecting for the difference between the model specification and ideal specification -- an instance of regressional Goodhart. But in practice, we don't get the model specification; instead humans choose some particular proxy specification, typically leading to good behavior on training environments. However, in new regimes, this may result in optimizing for some extreme state where the proxy specification no longer correlates with the model specification, leading to very poor performance according to the model specification -- an instance of extremal Goodhart. (Most of the classic worries of specifying utility functions, including e.g. negative side effects, fall into this category.) Then, we have to actually implement the proxy specification in code, giving an implementation specification. Reward tampering allows you to "hack" the implementation to get high reward, even though the proxy specification would not give high reward, an instance of causal Goodhart.

They also argue that the ideal -> model -> proxy problems are instances of problems with selection, while the proxy -> implementation problems are instances of control problems (see Selection vs Control (AN #58)). In addition, the ideal -> model -> proxy -> implementation problems correspond to outer alignment, while inner alignment is a part of the implementation -> revealed specification problem.

Technical agendas and prioritization

Useful Does Not Mean Secure (Ben Pace) (summarized by Rohin): Recently, I suggested the following broad model: The way you build things that are useful and do what you want is to understand how things work and put them together in a deliberate way. If you put things together randomly, they either won't work, or will have unintended side effects. Under this model, relative to doing nothing, it is net positive to improve our understanding of AI systems, e.g. via transparency tools, even if it means we build powerful AI systems sooner (which reduces the time we have to solve alignment).

This post presents a counterargument: while understanding helps us make useful systems, it need not help us build secure systems. We need security because that is the only way to get useful systems in the presence of powerful external optimization, and the whole point of AGI is to build systems that are more powerful optimizers than we are. If you take an already-useful AI system, and you "make it more powerful", this increases the intelligence of both the useful parts and the adversarial parts. At this point, the main point of failure is if the adversarial parts "win": you now have to be robust against adversaries, which is a security property, not a usefulness property.

Under this model, transparency work need not be helpful: if the transparency tools allow you to detect some kinds of bad cognition but not others, an adversary simply makes sure that all of its adversarial cognition is the kind you can't detect. Rohin's note: Or, if you use your transparency tools during training, you are selecting for models whose adversarial cognition is the kind you can't detect. Then, transparency tools could increase understanding and shorten the time to powerful AI systems, without improving security.

Rohin's opinion: I certainly agree that in the presence of powerful adversarial optimizers, you need security to get your system to do what you want. However, we can just not build powerful adversarial optimizers. My preferred solution is to make sure our AI systems are trying to do what we want, so that they never become adversarial in the first place. But if for some reason we can't do that, then we could make sure AI systems don't become too powerful, or not build them at all. It seems very weird to instead say "well, the AI system is going to be adversarial and way more powerful, let's figure out how to make it secure" -- that should be the last approach, if none of the other approaches work out. (More details in this comment.) Note that MIRI doesn't aim for security because they expect powerful adversarial optimization -- they aim for security because any optimization leads to extreme outcomes (AN #13). (More details in this comment.)


Verification and Transparency (Daniel Filan) (summarized by Rohin): This post points out that verification and transparency have similar goals. Transparency produces an artefact that allows the user to answer questions about the system under investigation (e.g. "why did the neural net predict that this was a tennis ball?"). Verification on the other hand allows the user to pose a question, and then automatically answers that question (e.g. "is there an adversarial example for this image?").

Critiques (Alignment)

We Shouldn’t be Scared by ‘Superintelligent A.I.’ (Melanie Mitchell) (summarized by Rohin): This review of Human Compatible (AN #69) argues that people worried about superintelligent AI are making a mistake by assuming that an AI system "could surpass the generality and flexibility of human intelligence while seamlessly retaining the speed, precision and programmability of a computer". It seems likely that human intelligence is strongly integrated, such that our emotions, desires, sense of autonomy, etc. are all necessary for intelligence, and so general intelligence can't be separated from so-called "irrational" biases. Since we know so little about what intelligence actually looks like, we don't yet have enough information to create AI policy for the real world.

Rohin's opinion: The only part of this review I disagree with is the title -- every sentence in the text seems quite reasonable. I in fact do not want policy that advocates for particular solutions now, precisely because it's not yet clear what the problem actually is. (More "field-building" type policy, such as increased investment in research, seems fine.)

The review never actually argues for its title -- you need some additional argument, such as "and therefore, we will never achieve superintelligence", or "and since superintelligent AI will be like humans, they will be aligned by default". For the first one, while I could believe that we'll never build ruthlessly goal-pursuing agents for the reasons outlined in the article, I'd be shocked if we couldn't build agents that were more intelligent than us. For the second one, I agree with the outside view argument presented in Human Compatible: while humans might be aligned with each other (debatable, but for now let's accept it), humans are certainly not aligned with gorillas. We don't have a strong reason to say that our situation with superintelligent AI will be different from the gorillas' situation with us. (Obviously, we get to design AI systems, while gorillas didn't design us, but this is only useful if we actually have an argument why our design for AI systems will avoid the gorilla problem, and so far we don't have such an argument.)

Miscellaneous (Alignment)

Strategic implications of AIs' ability to coordinate at low cost, for example by merging (Wei Dai) (summarized by Matthew): There are a number of differences between how humans cooperate and how hypothetical AI agents could cooperate, and these differences have important strategic implications for AI forecasting and safety. The first big implication is that AIs with explicit utility functions will be able to merge their values. This merging may have the effect of rendering laws and norms obsolete, since large conflicts would no longer occur. The second big implication is that our approaches to AI safety should preserve the ability for AIs to cooperate. This is because if AIs don't have the ability to cooperate, they might not be as effective, as they will be outcompeted by factions who can cooperate better.

Matthew's opinion: My usual starting point for future forecasting is to assume that AI won't alter any long term trends, and then update from there on the evidence. Most technologies haven't disrupted centuries-long trends in conflict resolution, which makes me hesitant to accept the first implication. Here, I think the biggest weakness in the argument is the assumption that powerful AIs should be described as having explicit utility functions. I still think that cooperation will be easier in the future, but it probably won't follow a radical departure from past trends.

Do Sufficiently Advanced Agents Use Logic? (Abram Demski) (summarized by Rohin): Current progress in ML suggests that it's quite important for agents to learn how to predict what's going to happen, even though ultimately we primarily care about the final performance. Similarly, it seems likely that the ability to use logic will be an important component of intelligence, even though it doesn't obviously directly contribute to final performance.

The main source of intuition is that in environments where data is scarce, agents should still be able to learn from the results of (logical) computations. For example, while it may take some data to learn the rules of chess, once you have learned them, it should take nothing but more thinking time to figure out how to play chess well. In game theory, the ability to think about similar games and learning from what "would" happen in those games seems quite powerful. When modeling both agents in a game this way, a single-shot game effectively becomes an iterated game (AN #25).

Rohin's opinion: Certainly the ability to think through hypothetical scenarios helps a lot, as recently demonstrated by MuZero (AN #75), and that alone is sufficient reason to expect advanced agents to use logic, or something like it. Another such intuition for me is that logic enables much better generalization, e.g. our grade-school algorithm for adding numbers is way better than algorithms learned by neural nets for adding numbers (which often fail to generalize to very long numbers).

Of course, the "logic" that advanced agents use could be learned rather than pre-specified, just as we humans use learned logic to reason about the world.

Other progress in AI   Reinforcement learning

Stabilizing Transformers for Reinforcement Learning (Emilio Parisotto et al) (summarized by Zach): Transformers have been incredibly successful in domains with sequential data. Naturally, one might expect transformers to be useful in partially observable RL problems. However, transformers have complex implementations making them difficult to use in an already challenging domain for learning. In this paper, the authors explore a novel transformer architecture they call Gated Transformer-XL (GTrXL) that can be used in the RL setting. The authors succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism located at key points in the submodules of the transformer. The new architecture is tested on DMlab-30, a suite of RL tasks including memory, and shows improvement over baseline transformer architectures and the neural computer architecture MERLIN. Furthermore, GTrXL learns faster and is more robust than a baseline transformer architecture.

Zach's opinion: This is one of those 'obvious' ideas that turns out to be very difficult to put into practice. I'm glad to see a paper like this simply because the authors do a good job at explaining why a naive execution of the transformer idea is bound to fail. Overall, the architecture seems to be a solid improvement over the TrXL variant. I'd be curious whether or not the architecture is also better in an NLP setting.

Copyright © 2019 Rohin Shah, All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.


2019 Winter Solstice Collection

Новости LessWrong.com - 4 декабря, 2019 - 20:25
Published on December 4, 2019 5:25 PM UTC

If you know of a 2019 Winter Solstice event, or solstice-adjacent megameetup, please post relevant links in a comment here.


"Fully" acausal trade

Новости LessWrong.com - 4 декабря, 2019 - 19:39
Published on December 4, 2019 4:39 PM UTC

Acausal trade happens when two agents manage to reach a deal with each other, despite not being able to interact causally (and, in some cases, not being sure the other one exists). Consider, for example, the prisoner's dilemma played against another copy of yourself, either in the next room or the next universe.

But those two situations are subtly different. If my copy is in the next room, then we will interact after we've reached our decision; if they're in the next universe, then we won't.

It might seem like a small difference, but my simple way of breaking acausal trade succeeds in the "next universe" situation, but fails in the "next room" situation.

So it would be good to distinguish the two cases. Since the terminology is well established, I'll call the "next universe" situation - where there are no interactions between the futures of the agents - to be "fully" acausal trade.


Рациональное додзё. Факторизация избеганий

События в Кочерге - 4 декабря, 2019 - 19:30
На встрече поговорим об избеганиях и потренируем навыки борьбы с избеганиями. Готовьте свои темы: чего вы избегаете, какие умения хотели бы приобрести, но давно откладываете, какие дела вам почему-то очень не нравится делать.

In which ways have you self-improved that made you feel bad for not having done it earlier?

Новости LessWrong.com - 4 декабря, 2019 - 15:33
Published on December 4, 2019 12:33 PM UTC

It can be a decision, a skill, a habit, etc.

Can be because the improvement was very valuable, obvious in insight, a moral imperative, or any other reason.


A letter on optimism about human progress

Новости LessWrong.com - 4 декабря, 2019 - 07:21
Published on December 4, 2019 4:21 AM UTC

This open letter was originally posted on Letter.wiki and is part of a longer conversation with Andrew Glover about sustainability and progress.

Dear Andrew,

Thanks for a thoughtful reply. Reading over it, it seems the biggest difference between us is in our expectations for the future, in a word, our optimism. You agree it would be nice to give everyone the luxuries that only the rich enjoy today, but that “it doesn't seem possible to do this.” In a similar vein, on the topic of energy resources, you say you're “not aware of any principle that says new energy sources will be discovered simply by virtue of humans applying their ingenuity.”

So let's talk about that principle.

Certainly there is no law of physics that mandates inexorable progress, on any one axis or even in aggregate. Progress is not automatic or inevitable.

But human ingenuity has been so successful in such a wide variety of areas that I think, on the grounds of history and philosophy, we are justified in drawing a general principle: all problems are solvable, given enough time, effort, and thinking. Or to quote David Deutsch from The Beginning of Infinity, “anything not forbidden by the laws of nature is achievable, given the right knowledge.”

To take the historical view first, think of all the problems humanity has solved, all the magic we've created, that seemed impossible until it was invented—not just in energy, but in every field.

Our farms make an abundance of produce spring from the ground—reliably, consistently, year after year, rain or shine, flood or drought, regardless of what weeds, pests, or disease may attack our crops. We do this in many parts of the world, with different terrain, weather patterns, and growing seasons. We have done this not just through soil, fertilizer, and irrigation, but by breeding better plants—taking command of the taxonomy of species itself. And when the food is ready, we keep it fresh while it is transported all over the world; produce now knows no season or country.

We have largely conquered infectious disease. Except in the parts of the world too poor for effective water sanitation or mosquito control (and the parts of California where children aren't vaccinated), infectious disease is rare, and usually curable. Less than 200 years ago, we didn't know what caused these diseases or how they spread; today we have identified the specific microorganism behind every major malady, and sequenced their genomes.

The way we travel, too, would seem miraculous to our ancestors from 1800. One of them might have lamented that “it would be nice to give every peasant a fine horse and carriage like the king, but it doesn't seem possible given constraints on resources.” But today almost everyone has access to transportation much faster, safer, and more comfortable than any royalty of old. And not just on land and sea: we have broken the bonds of Earth's gravity to soar with the birds, and higher—to the Moon, and even (through our robot servants) to other worlds. No fear of getting lost, either, with detailed maps of every mile of the globe, and satellites far overhead acting as cosmic lighthouses.

Modern factories are equally amazing, in historical perspective, churning out an incredible variety of cheap products made to exacting specifications that couldn't be matched by the greatest master craftsman working by hand. Before the mechanization of the textile industry starting in the 1700s, it was an incredible luxury to have an entire wardrobe full of colorful, stylish clothes, which you can afford to throw out if they get worn or stained. Just the thought of it may have seemed to people of that era akin to how an expensive mansion or a private jet seems to us today—but we found a way to give it to almost everyone.

And I barely need to remind you of the absolute wizardry of electronics. Even the telegraph was a breakthrough in its day; imagine what Morse, Bell, or Edison would think of the iPhone 11 Pro on a 5G network.

All that is just what we can do. Think further of what we know. The mysteries we have solved, the secrets of the universe unlocked! We know the structure of matter and the makeup of stars. We detect swirling black holes a billion light years away, and subatomic particles in our own backyard. We synthesize chemicals of our own design. We image individual molecules, and the insides of human beings. We read the very code of life. But before the Scientific Revolution, all of this seemed beyond the range of human comprehension, a domain seen only by God.

When the human mind understands galactic rotation, the periodic table, thermodynamics, electromagnetism, chemical enzymes, the structure of the cell, the evolution of species—when it has solved problems not only in energy, but in transportation, infrastructure, agriculture, medicine, materials and manufacturing, supply chains and logistics, communication and computation, finance and management—why do you think we can't learn the knowledge we need and solve the problems facing us today? How many more examples do you need to increase your confidence in human ingenuity?

Perhaps one could look at this incredible track record and count it a lucky historical accident, not to be repeated—if there were no deeper, philosophic way to understand how it came about and what could keep it going. But there is: human beings are, again in Deutsch's words, “universal explainers”. Reason, the conceptual faculty, gives us the upper hand in any contest, even though Nature starts out with the home-field advantage.

It's true that we can't see today how we might solve some of the problems we face. But this has always been true, and it's the nature of progress. Just because we don't know how we'll solve problems, doesn't mean we won't.

That, historically and philosophically, is why I'm optimistic. But surely you know these facts—so why, then, are you relatively uninspired about the future?


Symbiotic Wars

Новости LessWrong.com - 4 декабря, 2019 - 03:06
Published on December 4, 2019 12:06 AM UTC

A meme is a self-replicating pattern of information. Some change. Some survive. Some die out.

The most virulent memes often reproduce themselves via their own inverses. Talking about how Flat Earthers are wrong increases awareness of the Flat Earther meme. Increasing awareness of the Flat Earther meme equals proliferating the Flat Earther meme. Anti Flat Earthers[1] spawn Flat Earthers. Flat Earthers spawn Anti Flat Earthers. Every Anti Flat Earther contains a dormant Flat Earther meme.

A scholar is just a library's way of making another library.

— Daniel Dennett

In this way Flat Earthers and Anti Flat Earthers are different stages of a single meme's lifecycle. If the Anti Flat Earthers' objective is to spread the Anti Flat Earth meme then this information reverb benefits the Anti Flat Earthers.

Capitalism and Communism are two halves of the most successful meme in history, the Cold War. Capitalism was defined by its opposition to Communism. Communism is defined by its opposition to Capitalism. It doesn't matter that Communist states spread a twisted version of Capitalism and Capitalist states spread a twisted version of Communism. During the Cold War, the title of your economic ideology mattered more than than your real-world economy.

Fighting a meme makes it stronger. The best way to kill a meme is to ignore it.

This presents a collective action problem. If you tell everyone to ignore the X meme then you've told everyone about X thereby spreading the X meme. Attacking a meme wins you Pyrrhic victories. To kill a meme you have to make the meme irrelevant by transcending it.

The Cold War meme died when the divide between Capitalism and Communism ceased to be meaningful. The Chinese Communist Party manages the world's largest capitalist economy. By 1901 Bolshevik standards, NATO is an alliance socialist nations.

  1. "Anti Flat Earther" is a different meme from "physics". Anti Flat Earthers refute Flat Earthers' arguments. The physics meme ignores quibble. ↩︎


Long-lasting Effects of Suspensions?

Новости LessWrong.com - 3 декабря, 2019 - 23:40
Published on December 3, 2019 8:40 PM UTC

I recently read "The School to Prison Pipeline: Long-Run Impacts of School Suspensions on Adult Crime" (Bacher-Hicks et. al. 2019, pdf, via Rob Wiblin) which argues that a policy of suspending kids in middle school leads to more crime as an adult.

Specifically, they found that after controlling for a bunch of things, students who attended schools with 0.38 more suspensions per student per year were 20% more likely to be jailed as adults:

A one standard deviation increase in the estimated school effect increases the average annual number of days suspended per year by 0.38, a 16 percent increase. ... We find that students assigned a school with a 1 standard deviation higher suspension effect are about 3.2 percentage points more likely to have ever been arrested and 2.5 percentage points more likely to have ever been incarcerated, which correspond to an increase of 17 percent and 20 percent of their respective sample means.

This is a very surprising outcome: from a single suspension in three years they're 20% more likely to go to jail?

The authors look at the Charlotte-Mecklenburg school district, was ordered by the court to desegregate in the 1970s. In the early 2000s the court was convinced that busing wasn't needed anymore, and the district implemented a "School Choice Plan" for beginning of the 2002 year. Students were massively shuffled between the schools and, while this was generally not randomized, the authors describe it as a "natural experiment".

The idea is that if a student moves from school A to school B and you know how often students were suspended at both schools, then you can look at differences later in life and see how much of that is explained by the difference in suspension rates. They note:

A key concern is whether variation in "strictness" across schools arises from policy choices made by administrators versus underlying variation in school context. Our use of the boundary change partly addresses this concern, because we show that schools' conditional suspension rates remain highly correlated through the year of the boundary change, which provides a very large shock to school context. We also show that school effects on suspensions are unrelated to other measures of school quality, such as achievement growth, teacher turnover and peer characteristics. And: We also test directly for the importance of administrative discretion by exploiting a second source of variation - principal movement across schools. We find that conditional suspension rates change substantially when new principals enter and exit, and that principals' effects on suspensions in other schools predict suspensions in their current schools. While we ultimately cannot directly connect our estimates to concrete policy changes, the balance of the evidence suggests that principals exert considerable influence over school discipline and that our results cannot be explained by context alone. Here's an alternative model that fits this data, which I think is much more plausible. Grant that differences in conditional suspension rates are mostly caused by administrators' policy preferences, but figure that student behavior still plays a role. Then figure there are differences between the schools' cultures or populations that are not captured by the controls, and that these differences cause both (a) differences in the student-behavior portion of the suspension rate and (b) differences in adult incarceration rates. If suspensions themselves had no effects we would still see suspension appearing to cause higher incarceration rates later in life.

They refer to movement of principals between schools, which offers a way to test this. Classify principals by their suspension rates, and look at schools that had a principal change while keeping the student body constant. Ideally do this in school districts where the parents don't have a choice about which school their children attend, to remove the risk that the student population before and after the principal change is different in aggregate. Compare the adult outcomes of students just before the change to ones just after. While a principal could affect school culture in multiple ways and we would attribute the entire effect to suspensions, this would at least let us check whether the differences are coming from the administration.

This sort of problem, where there's some kind of effect outside what you control for, which leads you to find causation where there may not be any, is a major issue for value-added models (VAM) in general. "Do Value Added Models Add Value?" (Rothstein 2010, pdf) and "Teacher Effects on Student Achievement and Height" (Bitler et. al. 2019, pdf) are two good papers on this. The first shows that a VAM approach yields higher grades in later years causing higher grades in earlier years, while the second shows the same for teachers causing their students to be taller.

I continue to think we put way too much stock in complex correlational studies, but Bacher-Hicks is an illustration of the way the "natural experiment" label can be used even for things that aren't very experiment-like. It's not a coincidence that at my day job, with lots of money on the line, we run extensive randomized controlled trials and almost never make decisions based on correlational evidence. I would like to see a lot more actual randomization in things like which teachers or schools people are assigned to; this would be very helpful for understanding what actually has what effects.


(Reinventing wheels) Maybe our world has become more people-shaped.

Новости LessWrong.com - 3 декабря, 2019 - 23:23
Published on December 3, 2019 8:23 PM UTC

Let's stow our QM caps and pretend Democritus was right: Atoms turn out to be the fundamental, indivisible units of physical reality after all. Let's further pretend that some flavor of hard determininism is in play: There is nothing apart from the motions and interactions of the atoms that governs the way the Universe ambles through time.

Past, perhaps, the first Planck moment of existence, where we might be initializing all of the constants we'll need, our billiard-ball universe is quite amenable to explanations at the atomic level in the language of causality. Why is Atom A here, and not there, at time t? Because it was caused to be there by the actions of the other atoms, and due to certain properties of A itself, in the time leading up to t.

In theory, this means that any level of abstraction we build up from the atomic one should preserve that ability to be described causally. But the amount of computational power we would need to actually pull that off would be staggering, far, far more than we could possibly fit within 3 pounds of grey matter. So even starting from the most deterministic possible model, as agents within the system, we don't really have the ability to directly leverage that causality.

Instead, we are forced by our own limited resources to construct abstractions that are simple enough for us to reason about. These abstractions throw out a lot of detail! And when you throw away even a small amount of detail, you lose the clean isomorphism-to-reality that allows our earlier statement of causality to be preserved. When you're dealing with atoms, in the billiard ball world, you can always predict where they'll be if you have enough power; when you're dealing with aprroximations of atoms, you lose the "always".

So not only is the map not the territory, if you lose the territory, there is no way to perfectly accurately reconstruct it from the map alone. If you ever find yourself in that unenviable place, you'll have to make (dare I say it) aesthetic decisions in the reconstruction process.

And that's a weird thing to think about, not least because that isn't the case in mathematics. If you have a finite-bandwidth mathematical signal, you can perfectly reconstruct it from a finite number of details about it with the correct sampling conditions. That's just the most direct example. What about how we can use set theory to define the natural numbers, for example, 0 = null; 1 = {null} = {0}; 2 = {{null}, null} = {1, 0}; ... ? In fact, mathematics abounds with territories that can be perfectly reconstructed from very small maps - that's one of the most interesting things about it as a subject.

In other words, it feels awfully like causality should be a rare and fleeting thing, as rare as pouring a bottle of clover honey into your chamomile tea and having it form into a tiny little honey-fairy taking a nice bath.

And yet... All of that clashes with my very enjoyably lived experience that, on the day-to-day human level, causality actually seems to work pretty freaking fine for most of my decisions.

Why? What on Earth is the difference?

I have a boring but serviceable hypothesis: We've just been around long enough to make our world human-shaped enough to let this happen.

Look around you. Chances are ninety percent of the interesting/useful/beautiful/etc. objects around you were either designed by humans, or placed there strategically by them. We might only have 3 pounds of grey matter to work with, but we had the fantastic luck to have a big chunk of that grey matter go towards a really good abstraction of how other humans worked. That means we can perform second-, and even third-order abstractions on how to set things up for them so they will have an easier time. When we wrap this chunk around and use it on ourselves, we often come up with better ways to do things than if we just introspected directly.

At the dawn of human history any one of us might have had a one-in-a-thousand chance of making the correct abstraction of "seed + soil + water = food" ex nihilo. That's okay, because the chances of us being able to spread that discovery to others are much higher. As the centuries unfolded, our ability to efficiently abstract one another might have allowed something like a compound-interest effect to take hold as regards these normally quite rare discoveries of bubbles of causality in the natural world; now we sit here in 2019 with a world where most of the most inscrutable problems we face are much bigger than ourselves. We worry about AGI, but we don't worry about, say, how to make a new T shirt once our last one falls apart. Even if we did have to make a new one by hand, we have mechanisms in place to acquire that knowledge; facing down the problem with nothing at our holster but trial and error is a much scarier proposition.


Russian x-risks newsletter #2, fall 2019

Новости LessWrong.com - 3 декабря, 2019 - 19:54
Published on December 3, 2019 4:54 PM UTC

Russia could be seen as an x-risks wonderland, with exploding nuclear facilities and bioweapons storage sites, doomsday weapons, frequent asteroid impacts, Siberian volcanic traps, Arctic methane bomb, military AI programs, and crazy scientists wanting to drill to the Earth’s core. But there are also people who do their best trying to prevent global risks, and one of the most well-known is Petrov, who is remembered on 9/26. Russia can also provide much of the world with lessons in resilience; for example, many of its people, living in villages, can still supply themselves with food, water and energy without access to external sources.


Explosion in the virology center in Novosibirsk, 16 September 2019. The explosion was caused by a natural gas tank used in the renovation process. The fire was allegedly close to the storage facilities but didn’t directly affect them, so there was no leak of hazardous biological materials.

Why it is important? This seems to be a new type of possible catastrophic event, that has not been previously predicted, one that could produce a “multipandemic”—the simultaneous release of many deadly biological agents. A possible protective measure against new events of this type is the preservation of deadly agents in different places.

What is it? This center is one of two places in the world where live samples of smallpox are preserved. The State Scientific Center for Virology and Biotechnology (SSC WB), also known as "the Vector Center", has one of the most comprehensive collections of viruses in the world, including Ebola, Marburg hemorrhagic fever, severe acute respiratory syndrome (SARS), smallpox, and others. Created in 1974 near Novosibirsk, it was previously a closed institute that led the development of vaccines, as well as strategies for protection against bacteriological and biological weapons. In 1999, Valentin Yevstigneev, the head of the biological defense department of the Russian Ministry of Defense, said that they began to consider the Vector Center as "an industrial base for the production of offensive biological preparations" in the late 1980s. It was assumed that the strains of smallpox, tularemia, plague, anthrax and Ebola developed by the center would be placed in warheads. This work was curtailed in 1992, shortly after USSR ceased to exist.


In October of 2019, Russian president Vladimir Putin signed into law a new national AI strategy, the text of which includes a passage about the need for fundamental research to develop strong (universal) AI. This line was supported during the main Russian AI event, the conference AI Journey in November 2019, where Schmidhuber and Ben Goertzel spoke about AGI. Putin later joined the conference, who reiterated that those who control AI will control the world, and also mentioned his belief in the need to create strong AI. The main mind behind the conference was German Gref, who is a big AI fan and the director of the Russian bank Sberbank. During the conference, six major Russian tech companies signed an agreement to form something like a Russian variant of the Partnership for AI. There were some ritual words about “AI ethics” during the conference, but nothing was said about the AI alignment problem. More technical sessions about AGI also took place; there were five presenters on topics including the DeepPavlov project from MIPT, and AIXI modification from Occam. Computer scientists Alexey Potapov (not to be mistaken for Russian cryonist Alexey Potapov) and Alexander I. Panov (not to be mistaken for Russian SETI scientist and astronomer Alexander D. Panov) presented at the meeting.

The new Russian AI strategy requested funding of around 6.5 billion USD until 2024, which is not a terribly large budget if compared with the AI strategies of other countries.

Runaway global warming

Scientists from Tomsk detected the eruption of a “methane bomb” in the East Siberian sea, which resulted in methane atmospheric concentration 6–7 times above normal, representing the biggest methane leak for 45 years of observation. Similarly, thousands of lakes have erupted methane in Siberia and Alaska. Meanwhile, Russia has signed the Paris climate agreement as of 23 September. However, Greta Thunberg-inspired climate protests had turnouts of less than 200 people and experienced police crackdowns.

Why it is important? Russia’s Arctic shelf and tundra has a large amount of accumulated methane sequestered in the form of ice-methane clathrate underwater and in organic material under permafrost. Because of the polar amplification of global warming, most of the warming is happening in polar regions. Methane itself is a much more potent greenhouse than CO2, but behaves differently: it has a half-life of only seven years, but it has “high global warming potential of 72 (averaged over 20 years) or 25 (averaged over 100 years)” times that of CO2 (phyz.org). This means that if we account for a one-time strong eruption, its effects are even stronger in the first seven years. Taken together, these factors suggest the possibility of a strong positive feedback loop of uncertain magnitude, but the heavy tail of this uncertainty includes runaway global warming leading to a lifeless planet.


A Russian scientist suggested we should nuke asteroids that are on a collision course with Earth. However, while nukes in space could be used as a weapon against targets on Earth, they would be ineffective in asteroid deflection, as pieces of an asteroid would hit the Earth if it was intercepted at short notice or close range. This approach would not help against large, kilometer-size asteroids in any case. Observation and intervention to change the orbits of asteroids long before the risk of impact may be the best preventive measure.

Meanwhile, Russia declared that it will resume production of medium-range nuclear missiles after the US abandoned the treaty which banned these missiles. The main goal of the treaty was to prevent accidental nuclear war, as shorter-range missiles could reach Moscow from Western Europe in 5–7 minutes, and the decision-makers would have little time to evaluate the reality of the threat. However, the advent of sea-based, air-based and hypersonic missiles has eroded the definition of “medium range”, and the US has previously accused Russia of violating the treaty.

Previous newsletter: Summer 2019


Уличная эпистемология. Тренировка

События в Кочерге - 3 декабря, 2019 - 19:30
Уличная эпистемология – это особый способ ведения диалогов. Он позволяет исследовать любые убеждения, даже на самые взрывные темы, при этом не скатываясь в спор и позволяя собеседникам улучшать методы познания.

Mistake Versus Conflict Theory of Against Billionaire Philanthropy

Новости LessWrong.com - 3 декабря, 2019 - 16:20
Published on December 3, 2019 1:20 PM UTC

Response To (SlateStarCodex): Against Against Billionaire Philanthropy

I agree with all the central points in Scott Alexander’s Against Against Billionaire Philanthropy. I find his statements accurate and his arguments convincing. I have quibbles with specific details and criticisms of particular actions.

He and I disagree on much regarding the right ways to be effective, whether or not it is as an altruist. None of that has any bearing on his central points.

We violently agree that it is highly praiseworthy and net good for the world to use one’s resources in attempts to improve the world. And that if we criticize rather than praise such actions, we will get less of them.

We also violently agree that one should direct those resources towards where one believes they would do the most good, to the best one of one’s ability. One should not first giving those resources to an outside organization one does not control and which mostly does not use resources wisely or aim to make the world better, in the hopes that it can be convinced to use those resources wisely and aim to make the world better.

We again violently agree that privately directed efforts of wealthy individuals often do massive amounts of obvious good, on average are much more effective, and have some of the most epic wins of history to their names. Scott cites only the altruistic wins and effectiveness here, which I’d normally object to, but which in context I’ll allow.

And so on.

Where we disagree is why anyone is opposing billionaire philanthropy. 

We disagree that Scott’s post is a useful thing to write. I agree with everything he says, but expect it to convince less than zero people to support his position.

Scott laid out our disagreement in his post Conflict vs. Mistake.

Scott is a mistake theorist. That’s not our disagreement here.

Our disagreement is that he’s failing to model that his opponents here are all pure conflict theorists.

Because, come on. Read their quotes. Consider their arguments.

Remember Scott’s test from Conflict vs. Mistake (the Jacobite piece in question is about how communists ignore problems of public choice):

What would the conflict theorist argument against the Jacobite piece look like? Take a second to actually think about this. Is it similar to what I’m writing right now – an explanation of conflict vs. mistake theory, and a defense of how conflict theory actually describes the world better than mistake theory does?

No. It’s the Baffler’s article saying that public choice theory is racist, and if you believe it you’re a white supremacist. If this wasn’t your guess, you still don’t understand that conflict theorists aren’t mistake theorists who just have a different theory about what the mistake is. They’re not going to respond to your criticism by politely explaining why you’re incorrect.

I read Scott’s recent post as having exactly this confusion. There is no disagreement about what the mistake is. There are people who are opposed to billionaires, or who support higher taxes. There are people opposed to nerds or to thinking. There are people opposed to all private actions not under ‘democratic control’.  There are people who are opposed to action of any kind. 

There are also people who enjoy mocking people, and in context don’t care about much else. All they know is that as long as they ‘punch up’ they get a free pass to mock to their heart’s content.

Then there are those who realize there is scapegoating of people that the in-group dislikes, that this is the politically wise side to be on, and so they get on the scapegoat train for self-advancement and/or self-protection.

Scott on the other hand thinks it would be a mistake to even mention or consider such concepts as motivations, for which he cites his post Caution on Bias Arguments.

Caution is one thing. Sticking one’s head in the sand and ignoring most of what is going on is another.

One can be a mistake theorist, in the sense that one thinks that the best way to improve the world is to figure out and debate what is going on, and what actions, rules or virtues would cause what results, then implement the best solutions.

One cannot be an effective mistake theorist, without acknowledging that there are a lot of conflict theorists out there. The models that don’t include this fact get reality very wrong. If you use one of those models, your model doesn’t work. You get your causes and effects wrong. Your solutions therefore won’t work.

There already were approximately zero mistake theorists against billionaire philanthropy in general, even if many of them oppose particular implementations.

Thus, I expect the main response to Scott’s post to mainly be that people read it or hear about it or see a link to it, and notice that there are billionaires out there to criticize. That this is what we are doing next. That there is a developing consensus that it is politically wise and socially cool to be against billionaire philanthropy as a way of being against billionaires. They see an opportunity, and a new trend they must keep up with.

I expect a few people to notice the arguments and update in favor of billionaire philanthropy being better than they realized, but those people to be few, and that them tacking on an extra zero in the positive impact estimation column does not change their behavior much.

There were some anti-government arguments in the post, in the hopes that people will update their general world models and then propagate that update onto billionaire philanthropy. They may convince a few people to shift political positions, but less than if those arguments were presented in another context, because the context here is in support of billionaires. Those who do will probably still mostly fail to propagate the changes to the post’s central points.

Thus, I expect the post to backfire.


Searching Along the Trail of Crumbs

Новости LessWrong.com - 3 декабря, 2019 - 16:10
Published on December 3, 2019 1:10 PM UTC

There are two extraordinarily powerful things going on right now in Standard. One of them is Cavaliers, which I played in Twitch Rivals and wrote about here.

The other is the food engine, a core of cards with several possible ways to finish.

The Core Engine

The never-touch-this-actual ever core are these fifteen cards:

4 Witch’s Oven

3 Caldron Familiar

4 Trail of Crumbs

4 Gilded Goose

Duplicate Cats are often low impact, and risk draws that have too much air, so it’s not clear that you want the fourth copy.

If you have the cats but you’re not running green at all, then you’re running Mono-Black Sacrifice or Rakdos Sacrifice. Those are decks, but they’re focusing on a different game plan.

With this engine, you have fifteen one and two mana cards. Trading with any of them without giving you value is usually impossible. Most subsets of the list are continuous advantage engines. Together, these cards give you both good things to do in the early turns and a late game engine that can grind out anyone who doesn’t finish you off. That’s quite the combination.

They also give you a continuous supply of sacrifice triggers, death triggers and food tokens for you to build upon, and also a key artifact worth looking for, which gives you a lot of directions you can go. The rest of this article is about exploring the various options for the remaining 20-21 spell slots.

Instead of fighting over snowballing planeswalker activations, we are now fighting over snowballing Trail of Crumbs activations and who can put more cats into ovens each turn.

Wicked Wolf

The list of additional options starts with Wicked Wolf. People are increasingly moving away from Wicked Wolf. They need to slow down. Wicked Wolf is amazing once you have the twelve sources of food. It was amazing when we only had eight from Gilded Goose and Oko, Thief of Crowns.

Wicked Wolf gives you removal for almost all creatures in the format.

Wicked Wolf gives you an effectively indestructible threat to pressure opponents and planeswalkers. With so much food it can often go very large for a lethal attack.

Wicked Wolf gives you an additional zero-mana way to sacrifice food when you want to activate Trail of Crumbs.

Wicked Wolf with food available can hold the ground on its own or with minimal against a large percentage of aggressive strategies.

Wicked Wolf is the most reliable way to find cards that close the game once you have the engine running, taking down their problematic creatures then as an unstoppable attacker.

Wicked Wolf plays great Magic. Be very hesitant to play a food engine without it.

There are three arguments against it that I can see.

Case number one is that Wicked Wolf is that your build doesn’t want to support double green. The problem is that you already need to support first turn Gilded Goose. Once you are doing that, there is not much additional cost to supporting Wicked Wolf. Thus, for example, the Jund Sacrifice deck from Twitch Rivals that went undefeated on day one played only one Wicked Wolf, but had sixteen green sources from its lands plus eight from its creatures, so it still felt comfortable sideboarding Questing Beast. So I don’t think this holds water.

Case number two is that the card is bad against Jeskai (and control, where a parallel discussion will say basically all the same things). I think this is a pre versus post board confusion. In game two you want to fit in both Duress and access to removal for Fires, so your space is super tight, and you end up having other priorities. But it’s not like Wicked Wolf is bad even then. Dealing with it can be quite the challenge, it can take out a Cavalier, and it is very good against Sphinx of Foresight. It usually can’t be killed by any of the sweepers either. In game one I think you give up very little having this, as opposed to running three drops that die to Deafening Clarion or paying five mana for Massacre Girl, both of which hurt quite a bit. So I don’t think this matters much, either.

Case number three is that in the mirror Wicked Wolf is blocked forever by an endless stream of cats, and does not usefully block. That makes it a four mana removal spell that often requires eating a food, which is not impressive. That’s all true, but if they don’t have the full Cat-Oven trick online, then this puts meaningful pressure on them, and if you don’t have that trick, it prevents attacks that can potentially do a lot of damage. Even if they do have the full engine, you can often attack with multiple things, and increasingly there are ways to kill Witch’s Oven on both sides, which opens things up to attacks once again. So I see a lot of value in the creature after the fight, and the fight can take out Mayhem Devil or Gilded Goose or Massacre Girl and provide a lot of value. It’s certainly better than Murderous Rider, which dies when they sacrifice the target to Witch’s Oven, and you need some amount of removal. So again, I don’t see it.

And the pseudo-fourth argument is that the places it is best are aggro decks, which ‘don’t really exist.’ Except that they do exist.

Thus, I don’t get cheating on this one, at all. Even if you don’t maindeck all four copies, it’s one of the best cards against aggressive strategies, so I can’t see not having all four in the 75, and even where it’s bad, it’s not that bad. People sideboard it out in matchups where it can be surprisingly problematic to deal with, like against Cavaliers, where how many to have after board is a very interesting question I’m uncertain about.

From here I will assume we thus have a 19 card base rather than a 15 card base, which combined with the bare minimum 24 lands gives us 17 remaining slots to play with.

Land Base and Color Costs

We always have access to heavy green since we must support Gilded Goose and Trail of Crumbs. We always have access to at least some black so we can cast Caldron Familiar. So we get heavy green, heavy a second color, and if we want it enough, we get a splash color. In theory we could double (or even triple) splash using Paradise Druid and Fabled Passage plus Gilded Goose, if we wanted to do something wild.

If we run a two color build and stick to Golgari, our mana is solid. We get four Overgrown Tomb for free, and get additional dual lands from Fabled Passage and/or Temple of Malady. I do not like running Temples of any kind in these lists, because once you start using Trail of Crumbs every extra mana you get often matters on every turn of the game.

Fabled Passage also gives you four additional sources for any potential splash, and gives you four more triggers for Mayhem Devil.

I don’t think the deck requires four Fabled Passage if it is sticking to two colors, but it’s at worst small mistake to run four and make your mana that much more solid, especially if you run a 25th land. It also matters how invested you are in double black.

If you are running three colors, especially when that third color is red, then your splash is either something very tiny like black for three cats or it is a real third color. If it’s black for three cats, then you don’t want to run a Swamp and may not even want all eight shock lands, so Fabled Passage does not help you, and having all the extra shock lands makes another tapped land that much worse, so I think you avoid it. The same goes for if you’re splashing white for Ethereal Absolution and a few sideboard cards. If it’s a real third color, then I do not see a way out of running four copies of Fabled Passage and a basic land, plus shock lands, as your way to get the third color. The borderline case is red purely for Mayhem Devil. There I think that the extra Mayhem Devil trigger from Fabled Passage pushes you back to running four copies and the Mountain, even though that’s more red mana than you would otherwise need. If you also have Korvold, Fae-Cursed King, the decision is easy.

Four or even five colors are available for splashes, if you are willing to run Paradise Druid and Fabled Passage. That gives you 13 sources off one basic land, which is not so bad.

You can also run Castle Garenberg or Castle Locthwain.

Tapped green sources are scary with Gilded Goose, so you likely can’t play both four copies of Fabled Passage and also Castle Garenberg in the same build. Castle Garenberg gets you to Feasting Troll King if you are interested in that, and also allows you to make food while deploying Wicked Wolf, both a turn early, or to leave extra mana for Trail of Crumbs. I think two color builds should seriously consider choosing a Castle Garenberg as more valuable than a Fabled Passage, even without any six drops.

Castle Locthwain is a card you hope to never have to activate, but one does not always have Trail of Crumbs, and the price of including it is very low. The first copy is essentially mandatory, the second copy is much rarer, and this seems right.

Land counts range from 23 to 25. Running 23 is almost certainly wrong even without a high end. Whether to run 24 or 25 remains unclear to me, but these decks hate missing land drops and don’t want to have to dig for them with Trail of Crumbs. Versions that don’t have extra one or two drops beyond the core need the 25th land. Versions that run Paradise Druid or other cheap action should be fine at 24.

The limiting factor on green sources is first turn green, where you want at least 14. Beyond that you’re probably better off investing in shoring up other colors.

Green Options

The plausible additional green cards are Leafkin Elemental, Thrashing Brontodon, Lovestruck Beast, The Great Henge and Feasting Troll King. I’ve heard suggestion of End-Maze Forerunners or Nissa, Who Shakes the World, but those do not seem to address our actual needs. Return to Nature, Questing Beast and Shifting Ceratops are reasonable sideboard cards. Leafkin Elemental and Cavalier of Thorns are elementals that do relevant things, if that is relevant to your interests.

Paradise Druid

Paradise Druid is very common. I am skeptical. It gets caught up in Massacre Girl and Deafening Clarion. The key early spells are mostly one to three mana things, so it takes many turns for Paradise Druid to turn a profit. If you tap it on your turn it can get picked off when it matters. Your color can’t rely on it, especially since you should sideboard it out against Jeskai or other sweepers, and is good enough without it. It seems like Paradise Druid increases the number of bad things that can happen to you.

In exchange for that, you do get more acceleration to your high end to go with the color insurance, and something useful to do on turn two. The question is whether or not that is relevant to your interests. Casting a faster Feasting Troll King or Casualties of War, or a life-saving Massacre Girl, or even a quicker Korvold, Fae-Cursed King, can be worth a lot. The more high end you play, especially proactive high end, the more reasonable Paradise Druid becomes, but I’m still not excited.

A lot of players are more concerned than they should be that Massacre Girl won’t do her thing if you don’t have Paradise Druid. Between Caldron Familiar and Witch’s Oven you are usually fine, and Thrashing Brontodon can provide extra insurance if you need it. It seems rare that Massacre Girl won’t work and you care a lot that you didn’t have an enabler, as opposed to losing another card in the bloodbath.

Another phenomenon is that we mostly see either four copies or zero copies. Given how bad it is to draw two Paradise Druids, given that the second one stands a very strong chance of not even being worth playing, we should be seeing more builds with less copies. I can accept these lesser numbers as necessary for those not packing as many two drops and also having a more proactive high end. For people with more three drops, this seems like it mostly gets in the way.

Leafkin Elemental

If you are running Risen Reef, suddenly Leafkin Elemental becomes a quality card. Without Risen Reef, it does not do anything you want it to do that Paradise Druid does not do better, nor is there much call for a fifth copy. So see discussions of Risen Reef.

Thrashing Brontodon

This card is in a curiously good spot right now. Historically I think almost all people playing this card were making a mistake, where you either got an overpriced creature or an overpriced removal spell. Now, you still get that same combination, but it suddenly is strangely attractive to me because of what cards actually matter. If your opponent has Trail of Crumbs, Fires of Invention or Embercleave, then that card needs to die. It needs to die now. Overpaying to deal with it is acceptable. Same can sometimes go for Witch’s Oven. If your opponent has none of those cards, how are you losing this game, and is it so bad to pay for a 3/4 body to hold the ground for a bit given nothing bad is happening to you? Seems fine. You’d rather get a better deal but you’ll probably be fine.

That is the theory, in any case. Again as with Paradise Druid, we see players embracing the philosophy and playing the full four copies while other players play zero, when a mix of the two seems much better than the average value of the two options. They even work together.

It comes down to what role your build is trying to play. If your goal is to grind out the win over time, Thrashing Brontodon seems to be a great way to make that happen. If your goal is something more ambitious, this mostly gets in the way. Lists that differ by only a few cards answer that question remarkably differently. So it’s not obvious that people playing four or zero copies must be making a mistake, but I suspect that a lot of them should be compromising on copies more than they are. Your three slot is full of good options that get worse in multiples, and also Lovestruck Beast which improves in multiples.

Lovestruck Beast

I continue to strongly believe that in game one you either run four copies of this or you run zero copies, due to the need for more 1/1 creatures to allow your Lovestruck Beasts to attack. Drawing two does not even cause a curve problem. You get an incredible deal on power for cost, and the Caldron Familiars are the most reliable way yet found to keep a 1/1 creature on the board and allow you to attack.

The problem is that it is unclear that getting a cheap 5/5 creature does anything relevant for you at all, whereas you are passing up relevant other cards to get it. There’s no point in paying three mana in order to get blocked forever by a cat. Lovestruck Beast is great when it can block. It’s a solid clock. But if you don’t need Lovestruck Beast to hold the ground, and you don’t have a way to usefully attack with it, you’re falling behind on the engine race. So what is it good for?

There are two other things it is good for. There’s the minor bonus that Lovestruck Beast is good for three food in a Witch’s Oven, keeping your engine well-fed in a pinch. And there’s the advantage of putting a five-power creature onto the battlefield on turn three, which enables following it up with The Great Henge.

Cavalier of Thorns

There are a bunch of synergies and advantages here. The numbers allow for good fights against the more popular Cavaliers. Witch’s Oven allows you to kill off the Cavalier if you want to get back a card from your graveyard. Milling five cards helps you find your cats. It’s also an elemental for Risen Reef. Without that last effect, this is an interesting card, but having tried it the alternatives are too good and there isn’t enough here. It’s good in builds with Risen Reef, but that’s different from those builds being good decks.

The Great Henge

Now that this is no longer an Elk, and there are lots of games that go long and ways players keep themselves alive, this once again becomes an interesting Magic card. When it sticks around this tends to dominate the game, and the cat engine provides additional velocity so you never run out of gas. Decking yourself can actually be an issue that one has to think about during a game, to the extent that I’ve stopped using Witch’s Oven to avoid drawing the resulting cards, but those are not games one frequently loses.

The problem is having enough enablers, and having to play them. Wicked Wolf starts out at three power and can easily get to four or five, so it is at least reasonable, but it can’t get there on its own. Nothing else you naturally want to run that costs less than five is going to be good enough unless you go with Lovestruck Beast. Playing The Great Henge off a Feasting Troll King is a nice bonus option but not where you want any card’s default plan to be.

Rotting Regisaur is the best enabler of all, but it can’t be played in the main while this deck remains legal.

That forces you to run Lovestruck Beast as your primary enabler. Add in Wicked Wolf and something on the high end, and playing a miser’s copy of The Great Henge becomes reasonable. If you want to go bigger than that, the blue option of Vantress Gargoyle turbocharges things a lot, so you can check that out in the appropriate section. It’s a big commitment, but not without rewards.

The problem is that if opponents have Thrashing Brontodon and Casualties of War, then you’re doing a lot of work and often not getting much in return. New version of the Elk problem.

Feasting Troll King

Powering him out with Castle Garenberg is likely the best thing to do with five sources of mana these days. There is no response to that other than Planar Cleansing that doesn’t result in lots of value having been gained. This also benefits greatly from both enabling and being cast by The Great Henge, and provides a solid way to pull ahead when searching with Trail of Crumbs. Trample lets us cut through any cats in our way, and many opponents lack Wicked Wolf.

The problem is competing for the slot with Casualties of War, and to some extent various planeswalkers, and against the desire to keep the curve lower. This could be Garruk or Liliana. You can get a lot for six mana. This remains my favorite top end permanent to dig for, but all three options have their charms depending on what else you are up to.

Questing Beast

You can’t play Questing Beast main due to its vulnerability to Wicked Wolf and it not contributing to the core engine. In places where you need to put people on clocks while taking out planeswalkers, or its numbers are well positioned, it’s a crazy good Magic card. Right now it loses out to Shifting Ceratops, given exactly what else is out there, so you’d only play this if you wanted to devote 5+ slots of your board to this plan.

Shifting Ceratops

Shifting Ceratops in the abstract is a much worse card than Questing Beast, but it fills its particular role exceptionally well at the moment. Against Cavalier decks, along with packing a big punch it stops the air assault in its tracks and it can’t be bounced by Teferi, Time Raveler, so many very powerful boards have no answer. Its fifth point of power allows it to trade with Kenrith, the Returned King or Cavalier of Flame. This matters enough to consider modifying Cavalier builds to have good answers that don’t otherwise make sense to run. When facing Reclamation or Blue/White Control, the can’t be countered clause and the immunity from Brazen Borrower become important. So where this helps you, it’s a big upgrade, and thus a reasonable use of sideboard space.

Return to Nature

There are a bunch of enchantments and artifacts that have to die, and have to die now, and we’re bringing it in where we know we want it, so why pay 1GG and then 1 when you can pay 1G instead? One might even catch a cat in a graveyard if someone gets careless. The issues are that this is not a permanent, so we can’t find it with Trail of Crumbs, and also we can’t proactively deploy it to the table. I don’t think this justifies the space, but it is certainly reasonable.

Black or Golgari Options

Extending the use of black mana cuts off the opportunity to go deep into another color. Black as the secondary color is the natural move, since the deck relies on early cats and benefits from both Murderous Rider and Massacre Girl at double black as its best solutions to aggro. It is also excellent for your sideboard. The staple card is Midnight Rider. Then we can add Casualties of War, Garruk, Cursed Huntsman, Liliana, Dreadhorde Commander, Duress, Deathless Knight, Noxious Grasp, Leyline of the Void, Assassin’s Trophy and Rotting Regisaur.

Midnight Reaper

Your engine works much better with Midnight Reaper. Your creatures die a lot, and your cats die every turn while providing the life to compensate for the Reaper’s fee. It’s hard to not get some value for Midnight Reaper, and playing it greatly reduces the chance the engine will stall out. The question is how much you need this, whether to prioritize it over all the other great three drops, and how much to worry about its vulnerability. Being a three mana 3/2 is not a great place to be right now, giving opponents a juicy thing to target with Bonecrusher Giant and Mayhem Devil or to get caught up in Deafening Clarion. Deafening Clarion goes both ways, since that likely means drawing multiple cards, but you’d usually rather have had something that lived.

Consensus is to run two or three copies if you’re running a ‘normal’ Golgari or Jund build of the deck. That seems about right to me, as you do not need two copies but you’d usually like to have the first one.

Murderous Rider

Murderous Rider gives you Murder plus an extra creature while being a permanent for Trail of Crumbs. When the deck tries to do a second ambitious thing alongside the core food engine, it does so at the expense of having reliable removal. That reliable removal is the way one sideboards in many places, so not only does your sideboard end up with large pressure on its slots, your configuration after sideboarding ends up with the same issue. This is a big reward for keeping things straightforward. Playing four Murderous Rider and four Wicked Wolf wins a lot of games essentially on its own.

I was very happy to run four copies of Murderous Rider in several of my builds, and consider the card underplayed. If you are committed to playing a normal game of Magic and can support double black, the game currently offers nothing better. It doesn’t improve your engine, but where your engine needs the help it does help you break up the opposing engine. You sideboard out at least some copies in cat mirrors, but only because you have other cards that are more important.

The tax on the mana base is real. Murderous Rider wants you to pay 1BB twice, which is a much higher bar than any other plausible black card. Doing it twice means that Gilded Goose is not a good solution. Casualties of War and Massacre Girl are also double black, but they are more expensive and it’s usually fine to spend a food casting them if it comes to that.

Casualties of War

Casualties of War is rapidly growing in popularity because of its power against Jeskai Fires and in cat mirrors, which are the two most important matchups. Taking out Witch’s Oven, Trail of Crumbs, a land plus a creature together turns what looked like a perfect draw into a nightmare. Hitting Fires of Invention, a land and a creature is almost as good and challenges the viability of Sorcerous Spyglass out of Jeskai.

There are some places where Casualties of War is effectively a six mana creature removal spell, with the only other target being an irrelevant land. Then you’re sad you don’t have a more impactful high end play. Those are also the places where all the top end plays are bad except maybe Feasting Troll King, so it is only a problem for one game, and it costs you less than you might think since none of your other six drops were resolving and mattering all that often.

The natural objection is that Casualties of War is not a permanent, and a huge portion of your total seen cards are from Trail of Crumbs where you often are looking for big action more than anything else. That kept me off of Casualties of War at first. Then I realized that in the scenarios where you are digging deep into your deck, mana efficiency matters more than the quality of the cards chosen, because once you get started you can keep generating triggers as needed. Sure, you’d love to find Casualties of War, but you’re happy to be finding more one and two mana pieces instead. It’s fine to have it not get hit, so long as there aren’t too many other non-permanent cards in the deck, since missing or only finding unneeded lands can be pretty bad.

I consider Casualties of War to be the default six drop in versions without a secondary theme. This is especially true given how it lines up so well against the planeswalkers specifically and other food decks in general. I would only play other choices if I was doing something else intensive that needed specific help.

Liliana, Dreadhorde Commander

There are certainly games where Liliana wins that nothing else would have won. There are also a lot of games where my opponent plays Liliana, and the game changes surprisingly little. Or where I found myself thinking “I can beat anything except Liliana.” Overall I have still been underwhelmed by Liliana, in a variety of decks and a variety of matchups. Sure, the player who plays it usually wins, but it’s a six mana planeswalker, and most of those games would have been won by pretty much anything.

With so many copies of Murderous Rider and Casualties of War running around, and so many places where all this does is +1 and hope to eventually draw cards or ultimate, I don’t like playing her.

Garruk, Cursed Huntsman

Garruk certainly can feel, like Liliana, like the unique card that wins a game. You’ve dealt with the key permanent and then have a card that dominates the game, or you play Garruk and he’s better than the opponents’ entire deck without ever having to minus. There is certainly that, but more often he seems like he’s a lot of mana for something players often have a way to deal with even if it’s not terribly pretty. I buy the consensus that Liliana gets a slot before Garruk, and I don’t even want her to get one.


You want access to at least three Duress and I prefer four if the mana supports it. Duress is vital to beating Jeskai Cavaliers, Temur Reclamation and all the various control builds. Mana efficiency and protecting or preventing key cards is where it is at. The upgrade value of bringing it in where it is good is very large. It’s usually good even when it is drawn late. It might even be a great card against some cat builds if you have reliable first turn black, as it breaks up Trail of Crumbs and prevents Casualties of War. Playing a few copies in the main would not be crazy.

Noxious Grasp

Noxious Grasp is a great removal spell for decks that give you juicy targets to kill. If you can spare the sideboard slots, it’s certainly an upgrade where it is good. But it’s an effect you already have a bunch of to start with thanks to Murderous Rider, and none of the matchups where it is good seem especially popular or worrying. So while I would certainly be happy to have this available, I don’t feel that pressured to make room for it either.

Deathless Knight

You are gaining life all the time, so Deathless Knight will usually be truly deathless. This gives you four power with haste that can’t be permanently killed. Against control decks, this has to appeal, since the two toughness does not matter. I don’t feel any need to do this, because if there is one thing I have a lot of already it is ways to grind out card advantage. Every time I have had or seen someone try Deathless Knight, either it has sat in their hand because they had something better to do with their mana, or it has won games that most any card would have won in its place.

Rotting Regisaur

You absolutely cannot play Rotting Regisaur in the main of decks without Embercleave, given how easy it is to get brick walled by a cat and lose the game. That makes this a sideboard card even when you’re embracing The Great Henge. Where it is good, either where you want to hit harder against Temur Reclamation or have a big body against decks like red, it is very good even without The Great Henge, but it does not belong in the places you care about most. Is that worth the space? In ‘generic’ versions clearly no, since you have a lot of sideboard needs. In Henge-themed versions I think mostly yes, because you need to look for ways to sideboard that reinforce the central themes rather than dilute them.

Leyline of the Void

The attraction of sideboarding Leyline of the Void is obvious. For zero mana you shut down Midnight Reaper and Caldron Familiar, and Massacre Girl although that one is a double edged sword, and this is a permanent for Trail of Crumbs. Zero mana is a great deal.

The problem is that there are a ton of ways for this to go wrong.

You have to telegraph it before cards start getting eaten by the void. They can hold their cats in their hand if this starts in play. If it comes down they can often deposit them in the graveyard for safe keeping.

Then, once it is in play, they often can remove it when it matters and ignore it when it doesn’t. Leyline of the Void does not do that much work to stop the enemy from ramping to six and almost nothing to stop Thrashing Brontodon or Return to Nature.

Thus, I think you’re better off being the one with removal for enchantments and artifacts, rather than trying to run more of your own, given the importance of Trail of Crumbs. I don’t see any other places where Leyline justifies its space.

Assassin’s Trophy

Too many Jund bilds play this card for me to not mention it. I strongly dislike it. The land they get bites you back so often. This is especially true when you go after Fires of Invention. Almost always, when this kills a Fires of Invention, the extra land makes the loss of Fires much less painful, and often leaves them better off especially if Fires has already given them a free five drop or two. Players like it as a catch-all, since it can do what it needs to do. I do get that, but I don’t see any trouble getting to a good 60 against the whole field, so I don’t see any reason to go here.

Red Options

Red offers two key creatures in Mayhem Devil and Korvold, Fae-Cursed King. It also offers Cindervines. Then there is one other card worth discussing, which is Fires of Invention.

Mayhem Devil

Mayhem Devil is the reason to play Jund. It turbocharges everything the deck does, and works off the opponents’ triggers as well, which gets extremely painful quickly for other cat decks. If you have access to red, you are running four copies.

The downside is that a 3/3 for 1BR is exactly the wrong thing against Jeskai, where they have both Devout Decree and Deafening Clarion, and nothing important to kill, or other similar matchups where you would prefer something that made you resilient. But you can’t argue with the raw power on offer here, other than asking whether it is worth the pain and Fabled Passages.

Korvold, Fae-Cursed King

Korvold was designed as a commander. It shows. Leave Korvold alone and it creates lots of value for a deck that is focused on sacrifice triggers. Kill Korvold on the spot and you’ve likely invested five mana in order to go down on permanents. Korvold packs a big punch and flies but does not trample, and usually will have a window where it is vulnerable to Wicked Wolf. Overall I have not been impressed with it on either side of the table, as it seems like your high end card should not need to take this level of risk or this failure rate. It does pack a punch, so if I already had red mana access I don’t think playing one or two is unreasonable, but I’d do my best to avoid relying on this card.


Cindervines is a permanent that functions as a spell. If you would have considered Back to Nature, this is doing most of the things Back to Nature was doing while also getting in a bunch of damage. If you have access to red mana, this seems like by far the best way to answer problematic enchantments and artifacts, but it’s not a big enough swing to pull me towards playing red sources.

Fires of Invention

Fires of Invention will never be as good in a cat deck as it is in Jeskai Cavalier Fires, or in Grixis Planeswalker Fires. That is not the threshold for playing a Magic card. One can play Fires of Invention as a good card that makes your deck better, rather than a key card the deck is entirely built around. One does not even have to play all four copies.

Fires of Invention fixes your color entirely, and it lets you use your mana while casting two spells per turn. The food engine provides a bunch of plausible mana sinks while also providing the cards to cast multiple spells per turn, and games go long enough that the savings add up. When Fires of Invention gets together with Trail of Crumbs or a stream of Midnight Reaper triggers, it is a beautiful thing. In theory one could even tack on a Fae of Wishes engine, but that too is not necessary.

The problem is that if you have a four drop that only provides mana, you run serious risk of flooding, especially if you also run Paradise Druid, and if you build to take advantage of Fires of Invention you risk relying too much on having Fires of Invention. When this does work, one has to worry about it being a win more card rather than something that matters, since an active Trail of Crumbs engine plus action should usually be good enough to win the game anyway.

You can certainly do fun things with this card in cat decks.

White Options

White gives you a maindeck high end permanent with Ethereal Absolution, and quality flexible sideboard cards such as Kaya, Orzhov Usurper and Knight of Autumn. I have seen suggestion of Prison Realm, but that seems to me like a bad fifth Murderous Rider so I won’t say more about that one.

Ethereal Absolution

Ethereal Absolution dominates the board. Your creatures hit hard. Enemy cats stay dead, other enemies are severely weakened. Spirits are waiting if you are otherwise out of gas. Many decks have little or no chance once it hits. There is a lot to love.

The problem is that it is a six drop that often gives opponents a chance to remove it before the game snowballs fully out of control. If that response is Casualties of War, you’re screwed. Against other cat decks, if that response is Thrashing Brontodon or Return to Nature, you may have forced some awkwardness, but it is unlikely to have made that much difference. Thus, this is no longer a reliable trump card, so there isn’t that much attraction in jamming it, especially given the cost to the mana base, unless you have some strange trick on offer.

Kaya, Orzhov Usurper

Kaya, Orzhov Usurper is great at picking off cats and Witch’s Ovens, and doubles as a solid anti-aggression card or general solution to graveyards. You can’t main it as there are too many places where it does nothing relevant. She is certainly a welcome addition to the sideboard if white is reliably available, but isn’t enough better than alternatives to justify much of the price of that white access.

Knight of Autumn

If one is in the market for Thrashing Brontodon, you would think that surely one would be even more in the market for Knight of Autumn, but I’m no longer so sure. Having four toughness to survive Deafening Clarion, and to be proactive on the board for Embercleave, are both big issues of the Knight of Autumn, even with its additional utility options. Leaving behind a 2/1 body is not nothing, but it also does not matter much when both sides are fighting Trail of Crumbs wars. So gain, white has lost its luster.

Blue Options

Blue is the color of the artifact theme, with Emry, Lurker of the Lock and Vantress Gargoyle. It is the elemental color for Risen Reef, which can then bring along cards like Quasiduplicate and Agent of Treachery. It possibly also offers Fae of Wishes, if you’re a Fires of Invention kind of player.

Fae of Wishes

Fae of Wishes is great with Fires of Invention. If you have that engine going, you should win even if the rest of your cards are blank. Cat decks have an engine to provide extra card fuel and stall the game while this happens, so they are a fine place to consider putting this. The problem of course comes when you don’t have a Fires of Invention, at which point you mostly have a two mana 1/4 creature that does not do anything relevant. This means you have exacerbated the risk of drawing hands that don’t play properly slash the risk of having your plays broken up. I didn’t like Fae of Wishes in Cavalier Fires, and I don’t like it here either.

Vantress Gargoyle

Vantress Gargoyle provides a 5/4 flyer for 1U. That enables The Great Henge, and works great with Emry, Lurker of the Lock. Lowering the mana curve is great, and with four toughness this threatens to hit for a lot of damage eventually even if it spends a few turns not attacking. Milling cards helps you find your cats. You even can sideboard it out on the play (or, as I’ve considered in other decks, sideboard it in on the draw) when it won’t be able to properly block and can afford time for cards that cost more mana. Vantress Gargoyle is a super powerful Magic card that has not found the right home yet, with cats being yet another place that comes tantalizingly close.

Emry, Lurker of the Lock

Food is an artifact, so Emry more often than not can find a way to cost one mana. You get four cards closer to Witch’s Oven and to Caldron Familiar, along with any other artifacts you seek. The more artifacts you play, the better Emry gets. I found that Emry was quite good once you had Witch’s Oven, Vantress Gargoyle and The Great Henge. That forms a natural package. It does mean you run a substantial risk of missing, with Emry having nothing to target. We’d like to add a few copies of Golden Egg to fix that, if we can find the space and time. Having a few extra mana and food sources is far from the worst thing, but more durdling does seem like exactly what would not help matters. Thus, my inclination was to accept that Emry’s main job is to find key cards rather than provide an additional source of durdling.

Risen Reef

If you get to play a three mana card on turn two, Risen Reef seems like an excellent choice. That allows you to continue accelerating, it sets up a third turn playing Risen Reef or Leafkin Elemental, and generally is something no one without a ready-to-go Mayhem Devil or Deafening Clarion is going to be happy to see. We are focused on assembling our key cards in quantity and have a mana intensive engine, so anything that cycles us through the deck while deploying extra lands is certainly welcome. As discussed above, Leafkin Elemental and Cavailer of Throns both are also reasonable fits for the rest of what we are doing.

That gives us a twelve card base for our second theme to go with the fifteen card first theme, which then means about nine cards to fill out the deck and provide interaction. That has to pay for any Simic-style payoff cards like Agent of Treachery, Quasiduplicate or Jace, Wielder of Mysteries if you’re looking to go the full self-deck. It also has to pay for Wicked Wolf and any additional food engine components. Regular Simic builds already have the same issue where they need to devote so many slots to their engine. Combining the two makes things even worse.

Putting It All Together

That covers every card except Niv-Mizzet Reborn, which we’ll cover in the section where we build around it.

Thus, we can now can now tie a few of our options together: Golgari, Jund, Emry’s Restaurant, Fires of Niv-Mizzet and Cat Elementals. The first two are the standard strong builds of cats. The last three are some of my brews, which aren’t as good for now, but illustrate some other directions one can go. I do not believe Abzan is a viable approach at this time.


The gold standard for Golgari has to be Crokeyz’s build. He has been spearheading such strategies for a while, so his insights are all over this analysis. Here is the build he submitted for MC7:

2 Castle Locthwain
4 Casualties of War
3 Cauldron Familiar
3 Fabled Passage
10 Forest
4 Gilded Goose
2 Massacre Girl
3 Midnight Reaper
4 Murderous Rider
4 Overgrown Tomb
6 Swamp
3 Thrashing Brontodon
4 Trail of Crumbs
1 Vraska, Golgari Queen
3 Wicked Wolf
4 Witch’s Oven

2 Deathless Knight
4 Duress
1 Legion’s End
4 Lovestruck Beast
1 Massacre Girl
1 Noxious Grasp
2 Return to Nature

The core Golgari strategy is to defend against anything that can go over the top of you, then go over the top of them with Casualties of War combined with your engine. The innovation of Thrashing Brontodon gives you that needed protection. In general, you’re focused on playing as many solid cards as possible. Take care of your core needs, keep the mana excellent and minimize the chance anything bad happens.

As you would expect, I do not agree with all the choices above. Here are the places I disagree.

Fabled Passage seems to me like a tapped green source, whereas my long term green needs are not much higher than my first turn green needs, so running three copies seems like a lot. I’d certainly cut one of them for a Swamp and likely would keep cutting.

Legion’s End didn’t even get mentioned above because I do not know what good it is doing at the moment. It’s good against Edgewall Inkeeper I suppose, and helps cover you against strange creature rushes, but it seems entirely inessential.

Deathless Knight, as discussed above, has not impressed me and I’d rather have Shifting Ceratops in my board so I can also have it against Jeskai.

The fourth Wicked Wolf is a card he kept mocking people in his chat for wanting to cut, and then he cut it. I hereby mock him in turn, as I have no intention of letting it go. This may have been because of the nature of the tournament in question, in which case I do understand it, but I wouldn’t ladder without a full set.

That is about it, really. I’ve been on four Murderous Rider for a long time. I’m torn on moving the third Massacre Girl to the sideboard, but so is he, and again the nature of a Mythic Championship leans towards running less copies. I don’t actively have a problem with any of his choices.

My 75 would probably cut the Legion’s End and Deathless Knights, and add a Wicked Wolf and two copies of Shifting Certops, and that would be it.


Jund mainly gives you Mayhem Devil. One approach is to take Golgari, cut four flex slots for four Mayhem Devil, maybe one or two other cards for Korvold, Fae-Cursed King, and call it a day. Eight of your lands now shock you and one of them is colorless, in exchange you get an amazing creature. I do think Mayhem Devil is a substantial upgrade in those slots, but it does leave you more vunerable to Jeskai and isn’t something you can keep alive all that well in the mirror, so it’s hard to justify the price you must pay.

Most successful Jund players also use the aggressive posture of the red cards as a reason to play less engine cards and be more aggressive, including shifting their six drop towards planeswalkers and away from Casualties of War. They usually cut most copies of Wicked Wolf on the theory that they can use Mayhem Devil in that role, despite the two working together rather well. We frequently even see Assassin’s Trophy, a card I have always hated running. We see more copies of Vraska, Golgari Queen to get more sacrifice triggers.

I wonder how much of this is about the deck actually benefiting from those changes, and how much of it is that there is a play style and deckbuilding style that goes with playing Jund, and it causes players to make those choices, whereas players who are capable of giving up Mayhem Devil also choose to make different choices, plus a lot of deck copying that does not question such differences overly much.

Here’s a typical list, from the first Magic Online PTQ, by bnjy99, who finished 3rd:

2 Assassin’s Trophy
4 Blood Crypt
1 Castle Locthwain
4 Cauldron Familiar
3 Fabled Passage
6 Forest
2 Garruk, Cursed Huntsman
4 Gilded Goose
1 Korvold, Fae-Cursed King
1 Massacre Girl
4 Mayhem Devil
1 Midnight Reaper
4 Overgrown Tomb
4 Paradise Druid
4 Stomping Ground
3 Swamp
4 Trail of Crumbs
3 Vraska, Golgari Queen
1 Wicked Wolf
4 Witch’s Oven

2 Disfigure
4 Duress
1 Garruk, Cursed Huntsman
1 Lovestruck Beast
2 Noxious Grasp
3 Thrashing Brontodon
2 Wicked Wolf

Now for the fun stuff.

Emry’s Restaurant

If we take the core fifteen plus Wicked Wolf and combine it with Emry, Lurker of the Lock, Vantress Gargoyle and The Great Henge, plus Lovestruck Beast as an enabler for The Great Henge, we have two slots left. If we use them on Feasting Troll King, which is another logical progression of graveyards and The Great Henge, we get this:

4 Breeding Pool
2 Castle Garenbrig
3 Cauldron Familiar
4 Emry, Lurker of the Loch
2 Feasting Troll King
6 Forest
4 Gilded Goose
3 Island
4 Lovestruck Beast
4 Overgrown Tomb
1 Swamp
3 The Great Henge
4 Trail of Crumbs
4 Vantress Gargoyle
4 Watery Grave
4 Wicked Wolf
4 Witch’s Oven


4 Duress
1 Epic Downfall
3 Rotting Regisaur
3 Shifting Ceratops
2 Thrashing Brontodon
2 Vraska, Golgari Queen

This is a natural build. It takes good advantage of all of its cards, and needs all its pieces to interlock in order to tie itself together. Without access to other removal you definitely want four Wicked Wolf, so there are not many slots that can be challenged – you could cut one Feasting Troll King if you wish, or the fourth Emry, but if you cut pretty much anything else you might as well abandon the strategy entirely. That makes sideboarding difficult. One big advantage of Rotting Regisaur is that where you bring it in, it fills the role of ‘big power creature’ thus allowing you to cut Lovestruck Beast or Feasting Troll King or Vantress Gargoyle, depending on what you don’t wan in a given situation. The same is true for Shifting Ceratops. You need Vraska because you want a flexible card that answers Mayhem Devil, and your choices aren’t great.

I consider this a tier two build. It is lots of fun, it does powerful things, but it has the problem of many Simic decks that it does powerful things but has trouble turning doing powerful things into winning games.

Cat Elementals

The problem with Cat Elementals is that you are providing space for two engines at once. This does not leave much space for also interacting with the opponent and winning the game. Thus, when I tried out the following list…

4 Risen Reef (M20) 217
4 Witch’s Oven (ELD) 237
4 Cavalier of Thorns (M20) 167
3 Cauldron Familiar (ELD) 81
7 Forest (ELD) 269
4 Gilded Goose (ELD) 160
4 Trail of Crumbs (ELD) 179
4 Wicked Wolf (ELD) 181
3 Agent of Treachery (M20) 43
1 Castle Garenbrig (ELD) 240
2 Quasiduplicate (GRN) 51
4 Overgrown Tomb (GRN) 253
4 Watery Grave (GRN) 259
4 Breeding Pool (RNA) 246
4 Leafkin Druid (M20) 178
4 Island (WAR) 253


4 Lovestruck Beast (ELD) 165
3 Mystical Dispute (ELD) 58
3 Shifting Ceratops (M20) 194
2 Thrashing Brontodon (M20) 197
3 Duress (M19) 94

I ran into the most nightmarish board states I have ever seen. It did not help that I faced multiple other elemental decks, but the point was made regardless. Sam Black pointed out we could run a Jace, Wielder of Mysteries (and by implication, also a Tamiyo, Collector of Tales) if we wanted to in order to make decking ourselves a plan. That does seem like it would make effective sideboarding even harder, but perhaps it offers a path forward. We can also go deeper into the themes with Yarok, the Desecrated, as we have a lot of cards that trigger.

I sincerely hope this going along this path is not a good idea.

The Five Color Dragon: Niv-Mizzet Reborn

There is one more path, and I do not believe it’s top tier, but man is it a fun one.

We noticed that once we accept playing Fabled Passage and Paradise Druid, one basic land gets you thirteen sources of a splash color. The card you most want to cast is Casualties of War, which you currently can’t search for, and you also love a Mayhem Devil. Why not splash all five colors and play Niv-Mizzet?

Once I convinced myself the mana would work, I looked at all the gold cards in Standard to see what was worth fetching. We already have a good Golgari card with Casualties of War, so there isn’t much pressure in my mind to play Vraska, Golgari Queen. We also have Mayhem Devil. Korvold has an extra color, so it doesn’t count.

There weren’t many cards that were that appealing, but I realized that was fine. In the past, Niv-Mizzet Reborn decks have packed themselves full of gold cards to turn Niv-Mizzet into a draw-four or draw-five. But that’s completely unnecessary. If you can get Niv-Mizzet into play, what you care about most is the cards that want to follow Niv-Mizzet – the six drops. As long as you can find one of those, you still get your ideal sequence, and you’re still pulling way ahead, even if you don’t get a second card. If you can find even one more card, you’re good to go.

So I added another quality six drop to go with Casualties of War and Mayhem Devil, but decided that was enough. Let the rest of the deck be what it wants to be, throw in Fires of Invention for obvious reasons, and the deck builds itself:

3 Cauldron Familiar
2 Swamp
4 Witch’s Oven
1 Plains
1 Mountain
4 Trail of Crumbs
4 Mayhem Devil
1 Island
5 Forest
4 Casualties of War
4 Niv-Mizzet Reborn
2 Ethereal Absolution
4 Gilded Goose
4 Fabled Passage
4 Overgrown Tomb
2 Blood Crypt
4 Stomping Ground
3 Fires of Invention
4 Paradise Druid

3 Wicked Wolf
3 Duress
3 Lovestruck Beast
3 Cindervines
3 Massacre Girl

The sideboard is designed to ‘return you to normal’ where what you are doing is not relevant. The deck comes together naturally. You can slot in one card like Cindervines or Duress for disruption, but if you want to do more than that, then you cannot maintain both Niv-Mizzet all the payoffs and enablers that make Niv-Mizzet shine. Thus, you need to be prepared to pull much of the high end and ‘return to normal.’ The good news is that the normal setup is pretty great in those places, so having a few awkward lands and choices is not so bad. Otherwise, choose the tool that supplements what you are doing and otherwise stand pat.

Where do cats go from here? It would be surprising if Golgari and Jund cats do not remain a staple of the format. I consider them the strongest contender for title of ‘the best deck.’ Jeskai remains a better tool for dealing with random opponents and is a great deck, but a good player with a carefully built Golgari Cats deck can handle it.

The biggest question is, when you modify the Cat deck to be good in the mirror and against Jeskai, what does that open you up to? We won’t know the answer until enough players get far enough along that the opening becomes worth passing through.



Dance Weekend Schedule Checklist

Новости LessWrong.com - 3 декабря, 2019 - 04:50
Published on December 3, 2019 1:50 AM UTC

When I've been booked for a dance weekend, the organizers generally send a draft schedule out a few weeks in advance to look over. If they don't send one I'll usually ask about two weeks out. There are a few things I want to check, since it's easy for organizers to miss things when putting schedules together:

  • Is there enough time for sleeping from when we stop playing until we start playing the next morning, factoring in any time getting between housing and the venue? One weekend scheduled us to play until 11:30pm on Friday and start at 9:30am on Saturday, while housing us almost an hour from the hall.

  • Does the schedule include time for our initial sound check? Generally we want ~45min to get things right before playing the first time, ~30min minimum.

  • Does the schedule include time for changeovers? About 20min for one band to get off stage and the other to get on is usually good.

  • Are there any unnecessary band changeovers? As long as you leave enough time for sleeping, having one band end the evening and then start the next morning saves you a round of setting up the stage.

  • Are we playing about the amount of time we expected? One weekend had us playing four 3.5hr contra sessions which is really a lot.

  • If we're playing in multiple places, do we have enough time to get between them?

  • Are we scheduled to play anything we don't know how to play, or that needs special preparation? We're leading a sing-along? A slot says "techno contra" next to our name?

  • Are there any tweaks that would make naps or other traveling-with-a-kid things easier?

  • Are we playing a medley, and if so is it a reasonable length? One event scheduled us to play a 1hr15min medley; after talking with the organizers we ended up playing it ABA with the other band.

  • Are we getting to work with both callers? In a typical two-band two-caller two-night weekend there's no way to have all three of (a) both bands get a night of opening and a night of closing, (b) both callers get the same, and (c) both bands get a night slot with both callers. I'd rather lose (a) or (b) than (c), but it's especially sad when we don't end up working with one of the callers at all.

This isn't the same list everyone will have: some people need more sleep, others need to minimize the number of times they get on/off the stage, others play very poorly early in the morning or late at night, and others have restrictions I haven't thought of. While there are some issues that won't be clear until you've been there and see what it's like (the building has the AC set to 58 with chiller units right behind the band‽) it's worth it to look over the schedule carefully to notice places where adjustments could be helpful.


MIRI’s 2019 Fundraiser

Новости LessWrong.com - 3 декабря, 2019 - 04:16
Published on December 3, 2019 1:16 AM UTC

(Crossposted from the MIRI blog)

Giving Tuesday is tomorrow, December 3! Facebook will match up to $100,000 per organization, starting at 5:00:00am PT—so set those alarms early! The matching pool will likely be used up within 10 seconds, so see Colm’s post for tips on how to get donations to MIRI’s Fundraiser FB Page matched. Giving Tuesday donations will be included in the fundraiser total.

MIRI’s 2019 fundraiser is now live, December 2–31!

Over the past two years, huge donor support has helped us double the size of our AI alignment research team. Hitting our $1M fundraising goal this month will put us in a great position to continue our growth in 2020 and beyond, recruiting as many brilliant minds as possible to take on what appear to us to be the central technical obstacles to alignment.

Our fundraiser progress, updated in real time:

MIRI is a CS/math research group with a goal of understanding how to reliably “aim” future general-purpose AI systems at known goals. For an introduction to this research area, see Ensuring Smarter-Than-Human Intelligence Has A Positive Outcome and Risks from Learned Optimization in Advanced Machine Learning Systems. For background on how we approach the problem, see 2018 Update: Our New Research Directions and Embedded Agency.

At the end of 2017, we announced an expanded research focus and a plan to substantially grow our research team, with a goal of hiring “around ten new research staff over the next two years.” Two years later, I’m happy to report that we’re up eight research staff, and we have a ninth starting in February of next year, which will bring our total research team size to 20.[1]

We remain excited about our current research directions, and continue to feel that we could make progress on them more quickly by adding additional researchers and engineers to the team. As such, our main organizational priorities remain the same: push forward on our research directions, and grow the research team to accelerate our progress.

While we’re quite uncertain about how large we’ll ultimately want to grow, we plan to continue growing the research team at a similar rate over the next two years, and so expect to add around ten more research staff by the end of 2021.

Our projected budget for 2020 is $6.4M–$7.4M, with a point estimate of $6.8M,[2] up from around $6M this year.[3] In the mainline-growth scenario, we expect our budget to look something like this:

Looking further ahead, since staff salaries account for the vast majority of our expenses, I expect our spending to increase proportionately year-over-year while research team growth continues to be a priority.

Given our $6.8M budget for 2020, and the cash we currently have on hand, raising $1M in this fundraiser will put us in a great position for 2020. Hitting $1M positions us with cash reserves of 1.25–1.5 years going into 2020, which is exactly where we want to be to support ongoing hiring efforts and to provide the confidence we need to make and stand behind our salary and other financial commitments.

For more details on what we’ve been up to this year, and our plans for 2020, read on!

1. Workshops and scaling up

If you lived in a world that didn’t know calculus, but you knew something was missing, what general practices would have maximized your probability of coming up with it?

What if you didn’t start off knowing something was missing? Could you and some friends have gotten together and done research in a way that put you in a good position to notice it, to ask the right questions?

MIRI thinks that humanity is currently missing some of the core concepts and methods that AGI developers will need in order to align their systems down the road. We think we’ve found research paths that may help solve that problem, and good ways to rapidly improve our understanding via experiments; and we’re eager to add more researchers and engineers’ eyes and brains to the effort.

A significant portion of MIRI’s current work is in Haskell, and benefits from experience with functional programming and dependent type systems. More generally, if you’re a programmer who loves hunting for the most appropriate abstractions to fit some use case, developing clean concepts, making and then deploying elegant combinators, or audaciously trying to answer the deepest questions in computer science, then we think you should apply to work here, get to know us at a workshop, or reach out with questions.

As noted above, our research team is growing fast. The latest additions to the MIRI team include:

Evan Hubinger, a co-author on “Risks from Learned Optimization in Advanced Machine Learning Systems”. Evan previously designed the functional programming language Coconut, was an intern at OpenAI, and has done software engineering work at Google, Yelp, and Ripple.

Jeremy Schlatter, a software engineer who previously worked at Google and OpenAI. Some of the public projects Jeremy has contributed to include OpenAI’s Dota 2 bot and a popular (but now deprecated) debugger for the Go programming language.

Seraphina Nix, joining MIRI in February 2020. Seraphina graduates this month from Oberlin College with a major in mathematics and minors in computer science and physics. She has previously done research on ultra-lightweight dark matter candidates, deep reinforcement learning, and teaching neural networks to do high school mathematics.

Rafe Kennedy, who joins MIRI after working as an independent existential risk researcher at the Effective Altruism Hotel. Rafe previously worked at the data science startup NStack, and he holds an MPhysPhil from the University of Oxford in Physics & Philosophy.

MIRI’s hires and job trials are typically drawn from our 4.5-day, all-expense-paid AI Risk for Computer Scientists(AIRCS) workshop series.

Our workshop program is the best way we know of to bring promising talented individuals into what we think are useful trajectories towards being highly-contributing AI researchers and engineers. Having established an experience that participants love and that we believe to be highly valuable, we plan to continue experimenting with new versions of the workshop, and expect to run ten workshops over the course of 2020, up from eight this year.

These programs have led to a good number of new hires at MIRI as well as other AI safety organizations, and we find them valuable for everything from introducing talented outsiders to AI safety, to leveling up people who have been thinking about these issues for years.

If you’re interested in attending, apply here. If you have any questions, we highly encourage you to shoot Buck Shlegeris an email.

Our MIRI Summer Fellows Program plays a similar role for us, but is more targeted at mathematicians. We’re considering running MSFP in a shorter format in 2020. For any questions about MSFP, email Colm Ó Riain.

2. Research and write-ups

Our 2018 strategy update continues to be a great overview of where MIRI stands today, describing how we think about our research, laying out our case for working here, and explaining why most of our work currently isn’t public-facing.

Given the latter point, I’ll focus in this section on spotlighting what we’ve written up this past year, providing snapshots of some of the work individuals at MIRI are currently doing (without any intended implication that this is representative of the whole), and conveying some of our current broad impressions about how our research progress is going.

Some of our major write-ups and publications this year were:

  • Risks from Learned Optimization in Advanced Machine Learning Systems,” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. The process of generating this paper significantly clarified our own thinking, and informed Scott and Abram’s discussion of subsystem alignment in “Embedded Agency.”
    Scott views “Risks from Learned Optimization…” as being of comparable importance to “Embedded Agency” as exposition of key alignment difficulties, and we’ve been extremely happy about the new conversations and research that the field at large has produced to date in dialogue with the ideas in “Risks from Learned Optimization…”
  • Thoughts on Human Models, by Scott Garrabrant and DeepMind-based MIRI Research Associate Ramana Kumar, argues that the AI alignment research community should begin prioritizing “approaches that work well in the absence of human models.”
    The role of human models in alignment plans strikes us as one of the most important issues for MIRI and other research groups to wrestle with, and we’re generally interested in seeing what new approaches groups outside MIRI might come up with for leveraging AI for the common good in the absence of human models.
  • Cheating Death in Damascus,” by Nate Soares and Ben Levinstein. We presented this decision theory paper at the Formal Epistemology Workshop in 2017, but a lightly edited version has now been accepted to The Journal of Philosophy, previously voted the second highest-quality academic journal in philosophy.
  • The Alignment Research Field Guide, a very accessible and broadly applicable resource both for individual researchers and for groups getting off the ground.

Our other recent public writing includes an Effective Altruism Forum AMA with Buck Shlegeris, Abram Demski’s The Parable of Predict-O-Matic, and the many interesting outputs of the AI Alignment Writing Day we hosted toward the end of this year’s MIRI Summer Fellows Program.

Turning to our research team, last year we announced that prolific Haskell programmer Edward Kmett joined the MIRI team, freeing him up to do the thing he’s passionate about—improving the state of highly reliable (and simultaneously highly efficient) programming languages. MIRI Executive Director Nate Soares views this goal as very ambitious, though would feel better about the world if there existed programming languages that were both efficient and amenable to strong formal guarantees about their properties.

This year Edward moved to Berkeley to work more closely with the rest of the MIRI team. We’ve found it very helpful to have him around to provide ideas and contributions to our more engineering-oriented projects, helping give some amount of practical grounding to our work. Edward has also continued to be a huge help with recruiting through his connections in the functional programming and type theory world.

Meanwhile, our newest addition, Evan Hubinger, plans to continue working on solving inner alignment for amplification. Evan has outlined his research plans on the AI Alignment Forum, noting that relaxed adversarial training is a fairly up-to-date statement of his research agenda. Scott and other researchers at MIRI consider Evan’s work quite exciting, both in the context of amplification and in the context of other alignment approaches it might prove useful for.

Abram Demski is another MIRI researcher who has written up a large number of his research thoughts over the last year. Abram reports (fuller thoughts here) that he has moved away from a traditional decision-theoretic approach this year, and is now spending more time on learning-theoretic approaches, similar to MIRI Research Associate Vanessa Kosoy. Quoting Abram:

Around December 2018, I had a big update against the “classical decision-theory” mindset (in which learning and decision-making are viewed as separate problems), and towards taking a learning-theoretic approach. [… I have] made some attempts to communicate my update against UDT and toward learning-theoretic approaches, including this write-up. I talked to Daniel Kokotajlo about it, and he wrote The Commitment Races Problem, which I think captures a good chunk of it.

For her part, Vanessa’s recent work includes the paper “Delegative Reinforcement Learning: Learning to Avoid Traps with a Little Help,” which she presented at the ICLR 2019 SafeML workshop.

I’ll note again that the above are all snapshots of particular research directions various researchers at MIRI are pursuing, and don’t necessarily represent other researchers’ views or focus. As Buck recently noted, MIRI has a pretty flat management structure. We pride ourselves on minimizing bureaucracy, and on respecting the ability of our research staff to form their own inside-view models of the alignment problem and of what’s needed next to make progress. Nate recently expressed similar thoughts about how we do nondisclosure-by-default.

As a consequence, MIRI’s more math-oriented research especially tends to be dictated by individual models and research taste, without the expectation that everyone will share the same view of the problem.

Regarding his overall (very high-level) sense of how MIRI’s new research directions are progressing, Nate Soares reports:

Progress in 2019 has been slower than expected, but I have a sense of steady progress. In particular, my experience is one of steadily feeling less confused each week than the week before—of me and other researchers having difficulties that were preventing us from doing a thing we wanted to do, staring at them for hours, and then realizing that we’d been thinking wrongly about this or that, and coming away feeling markedly more like we know what’s going on.
An example of the kind of thing that causes us to feel like we’re making progress is that we’ll notice, “Aha, the right tool for thinking about all three of these apparently-dissimilar problems was order theory,” or something along those lines; and disparate pieces of frameworks will all turn out to be the same, and the relevant frameworks will become simpler, and we’ll be a little better able to think about a problem that I care about. This description is extremely abstract, but represents the flavor of what I mean by “steady progress” here, in the same vein as my writing last year about “deconfusion.”
Our hope is that enough of this kind of progress gives us a platform from which we can generate particular exciting results on core AI alignment obstacles, and I expect to see such results reasonably soon. To date, however, I have been disappointed by the amount of time that’s instead been spent on deconfusing myself and shoring up my frameworks; I previously expected to have more exciting results sooner.
In research of the kind we’re working on, it’s not uncommon for there to be years between sizeable results, though we should also expect to sometimes see cascades of surprisingly rapid progress, if we are indeed pushing in the right directions. My inside view of our ongoing work currently predicts that we’re on a productive track and should expect to see results we are more excited about before too long.

Our research progress, then, is slower than we had hoped, but the rate and quality of progress continues to be such that we consider this work very worthwhile, and we remain optimistic about our ability to convert further research staff hours into faster progress. At the same time, we are also (of course) looking for where our research bottlenecks are and how we can make our work more efficient, and we’re continuing to look for tweaks we can make that might boost our output further.

If things go well over the next few years—which seems likely but far from guaranteed—we’ll continue to find new ways of making progress on research threads we care a lot about, and continue finding ways to hire people to help make that happen.

Research staff expansion is our biggest source of expense growth, and by encouraging us to move faster on exciting hiring opportunities, donor support plays a key role in how we execute on our research agenda. Though the huge support we’ve received to date has put us in a solid position even at our new size, further donor support is a big help for us in continuing to grow. If you want to play a part in that, thank you.

Donate Now
  1. This number includes a new staff member who is currently doing a 6-month trial with us.
  2. These estimates were generated using a model similar to the one I used last year. For more details see our 2018 fundraiser post.
  3. This falls outside the $4.4M–$5.5M range I estimated in our 2018 fundraiser post, but is in line with the higher end of revised estimates we made internally in Q1 2019.


Open & Welcome Thread - December 2019

Новости LessWrong.com - 3 декабря, 2019 - 03:00
Published on December 3, 2019 12:00 AM UTC

  • If it’s worth saying, but not worth its own post, here's a place to put it.
  • And, if you are new to LessWrong, here's the place to introduce yourself.
    • Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.



Подписка на LessWrong на русском сбор новостей