Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 1 час 36 секунд назад

Formula for a Shortage

19 мая, 2022 - 22:30
Published on May 19, 2022 7:30 PM GMT

Previous coverage (not required in any way): Formula For Dying Babies

Great Formal Workups of Much of This (Scott Lincicome): America’s Infant Formula Crisis and the ‘Resiliency’ Mirage. And another from Derek Thompson at The Atlantic. If you want a shorter summary, both of those are excellent. Also, together they fight crime did a podcast.

Last week I chronicled that there is a shortage of baby formula, especially specialized baby formula, due to a combination of the same reasons that hold whenever there is a shortage of anything.

The playbook never changes. Restrict supply and subsidize demand.

Not only do we subsidize demand, we do so via the WIC in a way that forces market consolidation and inflexibility in support.

Supply is restricted to begin with in both classic ways, onerous regulatory requirements and outright protectionism via tariffs and quotas, even with those with whom we have free trade agreements. On top of that the FDA is involved.

If you’re familiar with such dynamics none of this is surprising or all that new. This is written more as a reference post for the future, and for those who are not intimately familiar with how such things work.

The memes come easily.

There are a few more I saved for later on.

I am going over this again, now that the full picture is clear and politicians have made various new insane statements, because the situation is so perfect. It’s terrible, in the sense that mothers are panicking and having trouble finding formula to keep their kids alive. I’m quite unhappy about it happening. What I mean is that this is the perfect example of a situation in which all the things our government likes to do combine to create a mysterious completely unnecessary shortage of a vital product via driving out most potential suppliers. Then those forces combine to prevent the problem from being fixed, and those responsible then blame capitalism and corporations for a problem they would have handled quite well if they’d been permitted to do so.

The short term crisis began with the Abbott plant in Michigan being shut down and staying shut down for months. It’s being reopened now, but it will take six to eight weeks to properly resume production and get it onto store shelves – the FDA expects the situation to improve over the next couple of months. That’s the good scenario.

FDA Delenda Est

So it is reasonable to start with asking what happened with this factory.

Why did it take months to reopen a baby formula factory after it was shut down?

One guess. That’s right. The FDA.

They make their priorities very clear, as illustrated in this letter sent to Cafe Hayek.

I’m not sure if the FDA spokesperson meant to be this honest in responding to the infant formula shortage:

“Our first and foremost priority is ensuring that any recalled product remains off the market and we are working with the U.S. Department of Agriculture and manufacturers to ensure that parents have access to alternative, safe infant formula,” the FDA said in a statement to Fox News Digital on Tuesday.

As Ian Fillmore put it when sharing the reference:

Rephrasing that only slightly, “While we are working to ensure parents have access to infant formula, our first and foremost priority is ensuring that any recalled product remains off the market.”  Exactly.  The FDA does not exist to get products on the market.  It exists to keep products off the market.  They have no idea how to get a product to market; that’s just not what they do.

When the FDA says they are ‘working to get products on the market’ it means they are hard at work keeping those same products off the market, but they are also considering doing slightly less of that at some point in the future.

No promises.

Here’s a more official, they-posted-this-on-purpose official response.

I have no idea where they got the numbers for this graph, but from all the reports of running around desperately looking for formula I have the feeling 79% is not properly describing the situation in question in any practical sense and that this metric was chosen and sculpted to make the problem looks less serious.

I mean if there’s a week that you don’t restock store shelves then there’s nothing on the shelves, but to say this right after a graph that says you’re not making progress, while everyone is screaming that things are bad, seems unlikely to be the smart tactical approach here.

This is the FDA’s message to us. “A product needed for babies to survive, whose supply we have restricted, is selling out and people can’t find any. If you see anyone charging a market-clearing price that keeps it in stock for those who need it most and encourages more supply, please report them right away.”

As one would expect, other suppliers are doing their best to step up, but it’s not easy to ramp up production quickly. I’m guessing being willing to spend more money might help? Perhaps they would be able to do a lot more if maybe, I don’t know, they were able to charge more money for their desperately needed products.

Crazy talk, I know.

The good news is that the plant in Michigan is indeed going to eventually reopen. There is an agreement to that effect Abbott reached with the FDA.

What can we do in the meantime?

The obvious solution when one supplier proves temporarily unavailable is to turn to others.

Thanks to the FDA, it is not that simple.

You see, we must be protected against European baby formula… labeling.

Including the fear that perhaps the label is in German, and how would a family know how to prepare the formula? And don’t say ‘slap a big sticker on it in English’ or ‘still seems way better than not having formula and watered-down isn’t that big a tragedy anyway’ but that’s all crazy talk.

To which this reasonable objection is raised.

Except, no, the whole issue that no, one does not simply waive labeling requirements, they are absolutely not going to do that even if that is literally the only issue. On the contrary, after months of not budging things got bad enough that they are going to somewhat easelabeling requirements temporarily, and I am pleasantly surprised they followed through on their promises to do everything possible by doing even that.

Since this is, in our reality, pretty much ‘everything possible’:

Under the new guidance, the FDA will decide on a case-by-case basis whether “to allow the introduction into interstate commerce (including importation) of infant formula that is safe and nutritionally adequate, but that may not comply with all statutory and regulatory requirements.” Formula manufacturers can submit safety and nutrition information to the FDA to be granted a reprieve from enforcement of these regulations.

“The extent to which we exercise enforcement discretion may vary,” states the new FDA guidance on baby formula:

For example, an infant formula whose label does not list the nutrients in the order required … would need an exercise of enforcement discretion regarding that particular labeling requirement, and FDA may determine that enforcement discretion is appropriate. In contrast, an infant formula whose level for a specific nutrient is below the minimum that we require or does not contain a specific nutrient we require might not be an appropriate candidate for enforcement discretion, especially if the low level or absence of the nutrient could present a safety issue for infants.

So what they are doing, in an emergency, is allowing, out of the kindness of their hearts, for manufacturers to apply for the ability to temporarily import their products once the FDA explicitly approves them, on a case-by-case basis. When the problem was something completely irrelevant like listing ingredients in the wrong order, the FDA plans to (eventually) approve such an application, which will be good until November. If the issue is a trivial argument over something other than labeling, well, tough.

Or, another way to put it is this.

If I was a European manufacturer, saw these rules, saw that even these ‘relaxations’ were temporary and knew I couldn’t charge more than the non-shortage market rates, would I apply for FDA approval here? No, I most definitely would not.

Labeling is, of course, not the only issue. There are quotas and tariffs, differing detail requirements, and once one is not going to be bringing a product to the American market there are going to be a bunch of detailed technical regulatory reasons why one does not simply bring it into market right away.

What does an (unusually sane, even) former FDA head think of the situation?

The problem, you see, is that we don’t have enough oversight. If we’re going to ensure a totally unfree government-run market perhaps we do need more than nine people. The government isn’t doing enough to fix the problems that it is directly causing, like encouraging new entrants. If you want WIC to encourage new entrants, perhaps let those in WIC simply choose which baby formula to buy, at least from among those FDA approved?

Is this all pure regulatory capture? Yes. Absolutely.

Could all of that be hand waived away in an instant if someone who thinks like I do was in charge of such decisions? Yes. Absolutely.

I am not in charge.

Of course, if I was in charge these rules wouldn’t be there in the first place. If formula is good enough for Canadians or Germans it’s good enough for Americans.

By all accounts, European formula is if anything superior to our own, yet even in an emergency causing major political embarrassment and potentially starving babies, we can’t even waive such rules temporarily.

Another good question is why the market is so concentrated that shutting down one plant threatens the whole supply.

Under normal circumstances, there being only four total manufacturers would be cause to worry about a monopoly and to wonder why we weren’t ensuring competition.

Whereas in addition to shutting the foreigners out almost entirely, which also has the effect of discouraging foreign capacity, the government is also directly responsible for the consolidation in the American formula industry, due to the WIC.

WIC and the Pre-Existing Market

WIC is the Special Supplemental Nutrition Program for Women, Infants and Children. It ‘provides free healthy food and services to low-income women and their children, including nutrition counseling, breastfeeding support, health education, and referrals to other programs and services.’

WIC is responsible for about half of all formula purchases.

The obvious response is that it would be far better to give parents money, and then let them decide for themselves what to buy, but given people hate giving out money, giving out in-kind goods and services seems far better than doing nothing. Not the most efficient option, but seems like a reasonable compromise.

The problem is that the government purchases its formula via statewide contracts, and thus severely restricts which formulas parents can buy. Which, given the FDA has to approve any formula sold, is kind of odd. If you win such a contract to be the one who captures this half of the market, your sales in the state jump 300% and you have a huge competitive advantage. If you can’t win those contracts, you can’t compete. Thus, forced consolidation to be able to compete for the contracts.

The advantage of doing that is that the government can drive a hard bargain and reduce the price, and does. Companies even take a loss on the WIC contract in order to get into prime position in the other half of the market. Which is good if you want the government to pay lower prices. Less good if you want a competitive market with lots of capacity. It’s a trade-off.

There is also, of course, the need to deal with all the exacting demands of the FDA.

Which is another reason to consolidate. FDA requirements impose large fixed costs to how you do business and also in ‘dealing’ directly with the FDA itself. Once again, small competitors can’t compete and potential new entrants stay away.

When you make success dependent on who can make a deal with the government, when it’s who you know and you need to deal with high fixed costs, insiders prosper at the expense of outsiders.

Thus, it is no surprise that things were already on shaky ground before the Abbott plant closed, although they then rapidly got worse.

I agree that this is an unacceptable level of industry concentration and the antitrust agencies should take a hard to look at what the protrust agencies are doing.

So essentially we went from ‘we basically don’t care what’s going on at the plant so long as there are no problems’ to ‘now that we have an issue we need to solve all of these problems that we wouldn’t otherwise even notice’ and that led to extensive negotiating around what needed to be done. Negotiating what needs to be done necessarily means sitting around not, until those negotiations finish, doing the things that end up ‘needing’ to be done.

If the regulations were of good form there wouldn’t need to be a ‘negotiation’ because there’d be very clear rules and nothing left to negotiate. If there is a negotiation, either the FDA is claiming the right to make arbitrary demands, there is no way to know when you’ve done what the FDA requires without getting them to say so, or Abbott actually can’t satisfy the requirements of a safe plant and the FDA is finding a way to let them operate anyway.

Of course, one could say that 40% of all formula being out of stock is a little bit bigger of a deal than the original problem and this problem could be settled by liability and reputational damage. We don’t live in that world.

The Biden Administration is, months in, suggesting that states ‘allow WIC recipients to buy whichever formula products are in stock’ and also this gem.

“And, USDA is urging states to relax their requirements that stores keep a certain amount of formula in stock. This will offer relief to retailers and allow companies to manage inventories to meet demand.

I can’t sell you the last item. If I did, I’d be out of stock.

Pro Trust Versus Anti Trust

You’ve intervened in the market to encourage consolidation and favor insiders over outsiders via onerous regulations, government contracts controlling half of all sales and effective bans on any imports. There’s a shortage. What will you do now?

Call the CEOs of the formula companies before Congress. That is definitely what they should be spending their time doing right now.

Also: Study this stunning lack of competition in the market, of course.

Sen. Tammy Duckworth (D-Ill.) on Thursday wrote to Federal Trade Commission Chair Lina Khan asking for a “wide-ranging study” of market dynamics and competition in the infant formula industry.

“Shedding light on the business practices and market conditions that led to sustained infant formula shortages will enable Congress and regulators to develop effective legislative and regulatory responses to strengthen the resiliency of the infant formula supply chain and prevent future infant formula shortages,” Duckworth wrote.

There’s business practices and then there’s market conditions. The business practices are largely in response to the market conditions.

That’s not to excuse Abbott’s safety failures, which of course should be investigated. But that would be true regardless of the shortages. The ‘business practices’ that matter here are the consolidation into four companies, which in turn is the result of the ‘market conditions’ the government created. Which, of course, are likely in turn largely caused by regulatory capture from the formula companies.

Protrust government policy went pretty far. In particular, we made sure under Trump to shoot down this Worthwhile Canadian Initiative. We don’t only want market power over the domestic market and to bar foreign producers from entry into America, we are sabotaging foreign markets and production as well (here’s a link to the document Fang refers to).

It would actually be pretty great if one actually did investigate the dynamics involved in more detail, and suggested solving the actual root causes of it. Alas, I have a feeling such an investigation would be ordered to treat the WIC structure and lack of possible imports (and our sabotage of foreign production as well) as givens rather than variables, and thus reach the conclusion that it’s those greedy corporations that care about profits rather than people causing all this trouble when they should instead… I don’t even know exactly what anyone would want to be expecting here. But presumably the demand wouldn’t be anything to make the situation better.

Still, someone really ought to do an investigation. Although a lot of people already did, and are happy to share the results, if anyone wants to listen.

Senator Mike Lee is proposing sensible practical actions, because it is just our luck that the senator most prominently proposing lifting stupid barriers to trade (he also proposed good legislation on the Dredge Act and Jones Act) is for very legitimate unrelated reasons also one of the most hated senators.

Then again, the other advocate of that I know about in the senate is Ted Cruz, which implies that the being hated thing isn’t a coincidence. A better front person would be nice.

Here’s what Lee proposes in The Formula Act:

We do have at least a house candidate. Here is Suraj Patel, who is running for Congress in my district and only narrowly lost last cycle, being remarkably reasonable in the New York Times and on Twitter about what we might do. I’m going to meet with him next week, cause this all seems great. Patel has even come out in favor of building more housing, so he is most definitely One Of Us. There are a number of other signs as well.

We Must Do More

What is the Biden Administration doing to address the situation?

For the first few months, it seems, they did essentially nothing. There was an inevitable crisis coming, because demand exceeded supply and existing stocks were being run down with no timetable to get the Abbott plant back. Biden waited until the shortages mounted up.

This is way way less bad than it is in an exponential growth situation like a pandemic. In a pandemic that doubles every few days, if you wait until you have a problem then, as we have found out multiple times, it is far too late to do most of the things that were previously worth doing.

In this case the problem grows in more linear fashion, so if most of the problem lies in the future there’s a very good chance there’s still time to prevent most of the damage.

I’d still like ‘we shut down one of the four sources of baby formula for an indefinite period’ to trigger some alarm bells?

That’s the kind of request the Biden administration would call ‘mind reading.’ Who could have predicted the shortage of baby formula? There’s this clip asking Pete Buttigieg about the quote that ‘if we’d been better mind readers I guess we could have done something earlier.’

Why exactly does this require a mind reader? Why couldn’t you have done something earlier, sir? Why is that an unreasonable standard?

One could indeed say ‘why did they cut the clip before Pete could answer?’ That link leads to the full clip.

Pete Buttigieg’s answer here is pretty great politics and reflects detailed knowledge of the situation. He’s pretty great at this, and he cares about details, and this is mostly the best possible version of the thing that it is. I don’t know how to give a better answer in Pete’s shoes. You love to see it, and it makes me wonder what Pete (who is the Secretary of Transportation and thus has zero actual authority over the situation) would do if he actually got to make decisions instead of defending the decisions of others.

I’m breaking the answer down in detail because it is worthy of that breakdown, and it illustrates the perspective so well.

Pete’s direct response is no, they’ve acted ‘from day one.’

He starts by saying they moved to create more flexibility in the WIC program, as noted above. That’s indeed a good first move, mitigating the damage from dumb rules, but anyone on the ball enough to know to do that should be able to do the math and know it will buy some time but won’t be enough on its own.

We are also considering other things, he says. We are increasing other domestic production and are ‘looking at imports.’

Increasing domestic production elsewhere is good, although the ‘watch out for price gouging’ stuff is definitely not helping speed things along on that front.

Alex Tabarrok at Marginal Revolution, while sighing at the thought of repeating all the same points as always, points to implicit price controls like fear of being called a price gouger or other backlash as a potential explanation for regional variation in the shortages. The shortages are extreme in some places and absent in others because prices can’t properly adjust. This seems right to me.

And I suppose looking at imports is better than not looking at imports.

Pete’s core argument is to go on offense and blame Abbott: “We are here because a company was not able to guarantee that its plant was safe.”

Yes. We are here because one company was not able to do this to the satisfaction of the FDA, the FDA has thus shut them down for months, and that was enough to trigger this level of problem. Which it really, really shouldn’t be. And even if it was enough, there were months in which to solve the problem, during which you ‘looked’ at imports.

The argument that ‘if someone else had done their job we wouldn’t be having this conversation’ can be important information but it is also not a valid excuse for it also being true that if you had done your job we wouldn’t be having this conversation.

This company, which ‘seems to have’ 40% market share (again, how did that happen?) ‘messed up’ and is ‘unable to confirm that a major plant is safe and free of contamination.’

Notice once again that language. They can’t guarantee or confirm that the plant is safe and free of contamination.

That’s very different from an assertion that the plant is unsafe. The argument isn’t over whether the plant is safe, it’s over how to demonstrate that the plant is safe to the FDA. Abbott wasn’t doing proper maintenance and safety work before, but given the scrutiny and liability regime they’re about to be under, they have plenty of incentive to get it right. The mindset difference here is important.

He then says ‘obviously the most important thing is to get that plant in Michigan up and running.’ If that was true, they’d be willing to compromise on proof of safety or even to some extent on real safety to get the plant started faster. So what looks like the top priority is enforcing a super-high direct safety standard on the plant while putting babies across the country at risk to do so. When you say as he does ‘it has to be up and running as soon as possible and it has to be safe’ then you have a trade-off you’re refusing to acknowledge. All of which is being used as a blame game here, to ensure that it’s all Abbott’s fault.

He then says this is the difference between a ‘supply’ problem and a ‘supply chain’ problem. That’s an important point, but only if you take ‘supply’ to mean ‘domestic supply’ because we’ve automatically ruled out other supply. If you consider that our ‘supply chain’ to get formula from Europe is we intercept it at the border and impound it, then we very much have a supply chain problem. One can solve a supply chain problem by having more supply available at more convenient parts of the chain, to take pressure off the chain. Or one can solve a supply problem by improving the supply chain, and using alternative sources of supply.

As I said, it’s actually a damn good answer given the circumstances. It’s technically accurate, it’s detailed and specific, it shows concrete useful steps being taken, it goes on the offensive against a blameworthy target. The only way to give a better answer would be to be doing a better job solving the problem. That’s not up to him.

Were they acting from day one, as they often claim? Let’s go to the administration’s own telling of the story to get the context of that ‘mind reading’ claim, and, well…

President Joe Biden on Friday defended the White House’s response to a months-long shortage of infant formula in the United States and dismissed the idea that his administration should have acted sooner, saying they “moved as quickly as a problem became apparent to us.”

Following remarks in the White House Rose Garden, Biden pushed back when asked if the administration should have reacted faster to shortages of baby formula nationwide, telling CNN’s Jeremy Diamond, “If we’d been better mind readers, I guess we could have.”

When did the problem ‘become apparent’ here? If the answer is ‘the day the plant shut down’ then it’s a very odd turn of phrase. If it’s later, then that’s that. If it’s earlier, then they should say so and I will agree their mind reading game is pretty good.

I also love the next line so much.

The President later said resolving the issue requires that the federal government “move with caution as well as speed,” adding that it was important to “make sure what we’re getting is, in fact, first-rate product.”

What’s an emergency? What’s a trade-off?

To summarize efforts after the problem became apparent (whenever that was), as the assistant White House press secretary tells us, they are here to tell us they are leaving no stone unturned.

No stone unturned. Official White House sources confirm that they have identified all the stones one might turn, and that they have turned them.

The link he offers says that they are working with retailers and one of the few remaining additional manufacturers to get product onto store shelves by ‘removing supply chain bottlenecks’ and other neat stuff like that. Command economy stuff. This does not seem like a problem that requires such solutions – if there is a lot of demand and not enough supply, and you can ramp up supply, you’ll ramp up supply. If the supply isn’t distributed correctly, I trust Target and Walmart to figure out how to fix that. It’s kind of their thing.

Then again, if you don’t want to simply pay for this to happen and also you treat any attempt to profit from the situation by charging market clearing prices as ‘price gouging’ then you might be locked into needing to coerce a response instead.

One might say there are rocks like ‘buy the thing you have a shortage of on the world market’ or ‘reduce regulatory barriers restricting the supply of the thing where you have a shortage’ and you might want to work harder on turning those. There’s also ‘take less than four months or so to even reach an agreement on what Abbott has to do to restart its factory.’

And of course as discussed below, there’s ‘encourage substitution out of formula for babies old enough for that’ but that’s a hard message to have to give since it means admitting there’s a problem, and that some people will have to go without. In some circumstances admitting you have a problem is the first step, but not here. It’s hard to ration things that people feel they need, which many of them actually do really need to have their babies not die, and for which you are thus understandably unwilling to raise prices.

And of course, when all else fails, you can always declare defense production act.

Farmer Hooch @CompanyHoochI will now never think of anything else but this for the DPA and I hate @scottlincicome for that. https://t.co/W0o8QGRN7f Fox News @FoxNewsBREAKING: Biden invokes Defense Production Act, ‘Operation Fly Formula’ to address shortage https://t.co/lIkKng0cfcMay 18th 20225 Likes

Operation Fly Formula is going to be flying in baby formula from overseas.

The White House said DOD will use its contracts with commercial air cargo lines, as it did to move materials during the early months of the COVID-19 pandemic, to transport products from manufacturing facilities abroad that have met Food and Drug Administration (FDA) safety standards. 

That’s in case anyone was wondering if we have a free market in baby formula. Also, that last line isn’t going anywhere, so…

Yeah, no. We are continuing to not do that.

Actually, you know what we are doing instead? We are continuing to intercept European baby formula that people try to import, and impounding it.

Then they celebrate, in ways that sound a lot like drug busts.

Have we considered offering to pay money for baby formula that meets European safety standards and then letting ordinary commercial businesses take care of the rest? Or even not stopping them when they try to do it? No, I didn’t think so. Defense production act.

So, why didn’t they act earlier? Here is one model, mentioned while discussing the administration needing to ask Congress for $28 million ‘for the baby formula shortage’ because they’re trying to solve it through central action rather than lifting supply constraints or fixing rule sand incentives, and almost all the Republicans voting no.

It’s that second one. They assumed months ago that they couldn’t be blamed for something that ‘isn’t their fault.’

So two things to know.

One, if you’re the President, you get blamed for everything, no matter what. It doesn’t matter if it is ‘your fault.’ And this is why. Regular people don’t have the ability to differentiate what is and isn’t ‘your fault.’ Thus, when they can’t buy baby formula or there’s lots of inflation, if you’re in office? Congratulations, it’s your fault, and they’re going to punish you accordingly.

This is indeed correct strategy. First best is to have a better model, but this is not a reasonable or realistic ask. Barring that, it is good to go purely on outcomes. Yes, sometimes there will be blame that isn’t deserved. But much better to have the right incentives going forward. This illustrates that. If you’re the Biden administration, you shouldn’t not do anything because the problem isn’t your fault. It’s still on you to fix it. If you don’t want the job of fixing a bunch of stuff that isn’t your fault then public service is not right for you.

Second, this is totally their fault. Not in the sense that it isn’t the fault of the FDA or Abbott or our trade policy, and not in the sense that they actively made any of those things worse. There’s certainly plenty of other blame. But in the sense that they had the power to do something about it, including after the Abbott plant closed, in plenty of time, completely painlessly, and didn’t do so? Yeah. That.

The Starving Babies Caucus

No matter what else you say about the Biden Administration, I am confident they are not in favor of literally starving babies to death. They’re not as fully committed into the anti-babies-starving-to-death camp as I am, but if there was a knob that said ‘starve [adjective] babies to death’ I am confident that no matter what the adjective was they would turn the dial as far as they could in the ‘don’t starve any babies’ direction.

And I am confident that if they had the option on whether or not to starve any particular baby to death, they would choose not to do that. I believe they would do so even if it meant they would ruin their expensive coat. They might not go about it the way I would, but at least there’s that.

Not everyone is on that page.

When we ship pallets of baby formula to the southern border, we are giving that formula to babies that we are detaining. If we do not give those babies formula, then those babies will die.

Thus, this is an explicit call to starve certain specific [adjective] babies to death.

One could reasonably object that this framing is not fair. No, you would say, this is not the pro-starvation position. This is merely the insufficiently-anti-starvation position. It says that other priorities come first, that’s all. We’re very sorry for your loss.

And yes, that’s fair. We cannot, from the evidence presented, differentiate the pro-[adjective]-baby-starvation from the insufficiently-anti-[adjective]-baby-starvation positions. One could even argue that it’s clearly the second one, because when we did have enough baby formula for everyone (wow, the sentences one types these days) they were continuing to support shipping baby formula sufficient for the needs of the babies in question. So I suppose, again, there is that.

It is still, as they say, quite the take, and quite the political strategy.

Here’s Fox News with a segment taking the same point of view. Contra to the source I wouldn’t call this perspective ‘maximally evil’ due to having had two years to expand my imagination in such matters. That clip does have an interesting unrelated assumption in it – that if Psaki’s claim that they were ‘working on the problem for months’ were true that there then wouldn’t be a shortage. Yet we should consider that perhaps that is not actually true. That the ways this type of thinking is allowed to ‘work on’ the problem are not capable of solving the problem within a few months, or at least incapable of it until the situation reaches crisis levels. That is what my model outputs here.

Here is a statement from the Governor of Texas.

So basically this.

What I’d like to see is a call for us to buy formula from Mexico and use that formula at the southern border.

Because the logical knots this ties various people into is hilarious. Mexico, you see, does not restrict imports of baby formula, so they are fully stocked and could easily handle our orders, and their babies seem to do fine.

Practical Considerations

It’s a scary time to have an infant. Luckily the FDA is here to help with a fact sheet on how to deal with the shortages that the FDA created.

The official recommendations aren’t wrong but could, perhaps, be more helpful.

I would indeed try all of those things, but yes, not helping all that much, thanks.

I would totally make the phone call to ask the pediatrician, but the ‘list of things you must do otherwise you’ll be the one we blame’ thing is very much a central thing going on here and throughout everything to do with medicine and/or kids. If you haven’t done everything on our checklist, no matter how inapplicable or dumb, we’re ignoring your complaints and it’s all your fault.

If you did somehow get your hands on some of that illegal European formula, it’s worth noting that not only is it safe, there’s a good chance it’s actively better.

From the LittleBundle post:

While both agencies have requirements for what’s included in a formula, the European Commission has stricter guidelines about what sources can be used to meet these requirements. They also regulate additional inclusions, exclusions, allowances, formula stages, and certifications.

Nutrient Sources: All infant formulas include carbohydrates for energy and these carbohydrates are typically pulled from sugars and starches. The European Commission has a small list of allowable sources for carbohydrates which include lactose, maltose, glucose, maltodextrins, glucose syrup, pre-cooked starch, and gelatinized starch. Notably missing from this list is corn syrup and corn syrup solids, which are often the primary source of carbohydrates in US baby formulas. Additionally, the European Commission requires that at least 30% of carbohydrates in infant formula come from lactose, as this is the primary carbohydrate source in breast milk. The US does not have this same requirement, and some US formulas contain little or no lactose at all. 

You can read the whole thing for several other reasons why Europeans would want to keep away from dangerous American formula far more than Americans would have any reason to fear European labeling mistakes.

If you read more in that thread, there’s discussion of a NY Times article about why it would be dangerous to allow European baby formula, and it’s… really something. The insane troll logic is strong. He calls it ‘pure insanity’ and he’s right. Click through to the thread if you want the full story.

Here’s Perry Metzger also pointing out that the whole thing is pure regulatory capture, with the FDA doing the work of excluding the competition so that more profits can be made. None of this is a coincidence.

I kind of wish I didn’t know about the superiority of European formula, since it now means I have to feel a non-zero amount of bad for not finding a way to get European baby formula.

In a sufficiently dire emergency, one potential action is making one’s own formula. That used to be a reasonably standard thing to do.

If you’re looking to make your own formula, I don’t have a recipe for you, but I do know that Facebook is doing its part by censoring at least one vintage recipe largely because it contains the ‘disputed ingredient’ of Karo. Which is corn syrup. Which we in America still use in baby formula. Market and speech might not be so free these days.

But also because it hasn’t been ‘evaluated by governing authorities’ and as we all know, saying you can do X to create Y is false if experts haven’t verified your claim.

There are a bunch of websites that suggest what to do if you need to make your own formula. One thread suggested this one, another this one, both of which I have completely not checked because the point here is that such solutions existed in the past and haven’t stopped physically working. Please please do not make formula yourself based on a link you find here, if you must go down that road (and you totally don’t need to do that especially if you’re reading this) you need to do your own research on that and find people who understand that angle. I have no idea what the right answers are here and I’m not trying to figure it out either.

As it turns out, you know what else you can give to a ten month old if they have the good fortune to be Canadian? Milk.

The American Academy of Pediatrics got this one right, to their credit, but that’s not the way things are being framed so word does not seem to be getting out very well.

Also in many cases solid foods. There are a bunch of options. If we have a genuine shortage of baby formula, the rules that are currently focused on things like ‘simple guideline people can understand and not stress out about it’ give way to practical considerations. As Matt points out, if you switch out of formula at 9 months instead of 12 months, you save a lot more than 25% of formula you were using, because larger babies need more formula.

For younger babies, that won’t work.

It would work if others substituted out to free up what’s available, but what parents are going to go to emergency rations in order to take one for the team, especially if you’re not willing to be loud about the situation? Not many, even of the ones who know the situation.

Of these four options, the obvious one to solve is the second one. If one were to, say, allow parents to exchange their WIC vouchers for money then they could use that money to pay Amazon. I know, I’m crazy.

As for alternative solutions, the FDA’s job is to stop people from doing things, take one more in the replies.

Once again, sure, we’ve got a potential partial solution (although not a very practical one compared to the others, I don’t see how this scales enough to have much impact even if it worked fine), but we can’t prove we have a solution to the satisfaction of regulators in a way that anyone can bankroll.

When you’ve got all these rules in place about what you can’t do, when everyone has to ask permission for everything, you’ve destroyed your flexibility. Through the ages families and villages have muddled through when times got tough, but we’ve put a hard ban on muddling through at scale and a softer but still pretty forceful ban on muddling through without scale.

Which is somewhat helpful under normal circumstances, but if things get sufficiently muddled that you need to be muddling through them then it’s a real problem.

If you have an infant, as I do, should you be worried about not being able to buy formula?

Thanks to implicit price controls, it’s an all-or-nothing situation. You don’t have to worry that the price will rise. You do have to worry about not finding any at all.

My current sense is that if you’re able to go on Amazon, pay retail price without WIC and use one of the standard formulas, there’s no good reason not to keep a bigger buffer in place but you’re not going to have any trouble.

However, if you need a special formula, you are potentially in quite a lot of trouble, and I would absolutely be panicking if that were the case.

If you don’t need something special but do need to rely on WIC, which limits your purchases, it’s somewhere in the middle. Chances are it will all be fine, but those chances are not 100% and there are a few months before things clear up (even assuming things go smoothly from here, which they probably will.)

Conclusion: What Now?

On a personal level the next steps are clear enough. On a societal level, we’re going to get a bunch more of the blame game, and a bunch more of symbolic actions that don’t do that much while the important regulatory barriers remain in place, until a few months from now the shortages probably and hopefully clear up.

There will be various ‘investigations’ that will likely cause the market to become even more fragile. Thanks to the power of the dairy industry and the universal appeal of protectionism and ‘regulation’ of those things too important to allow their markets to properly function, I don’t expect any serious movement towards things improving.

If there’s a much worse crisis down the line and we’re looking at babies actually dying of starvation, maybe we’ll waive some pointless regulations. We ran the test with Covid and mostly went with the ‘let people die’ plan, but the timing is better here and hopefully babies get more consideration. You got to keep hope alive.

Everything Everywhere All At Once

The goal of all this was to put down, while things were fresh, a record of everything involved here, as an example to draw upon in the future.

The point is not that the formula market is uniquely screwed up. I don’t agree with statements that this was a ‘perfect storm’ of problems. Yes, all the classic problems showed up, but that’s the point. They usually show up. This situation is not the exception, this situation is the rule.

I talk about some problems being ‘too important to solve.’ Similarly, some goods are too vital to robustly or efficiently supply. The more vital the good, like baby formula, the more we mess with things, the more screwed up things get and the more fragile and dysfunctional the results we get.

We don’t have a baby formula shortage because we don’t value baby formula. We have a baby formula shortage because we value baby formula.

Generalize.



Discuss

How has the total amount of gain-of-function research worldwide grown/shrunk over time?

19 мая, 2022 - 18:57
Published on May 19, 2022 3:57 PM GMT

Given the advance in underlying biological tools/technology, I would not find it surprising if gain-of-function research had increased 10x over the last decade. (This is just based on priors; getting an actual estimate is the point of this question.)

Why is this interesting? Assuming that both original covid and at least one strain were lab leaks, the rate of growth of gain-of-function research has a huge impact on how often we should expect future pandemics from lab leaks. If the amount of gain-of-function research has been roughly constant over the last 20 years, then the two covid leaks suggest a base rate of something like one or two pandemic-level leaks every 20 years, and covid was probably unusually hard to contain in a lab for some reason on top of that. On the other hand, if the field has grown 10x in the past 5 years, that suggests a base rate more like a leak every 2-3 years (and probably getting faster over time as the field continues to grow). If that's the case, then the base rate for pandemics will be way higher going forward than in the past.



Discuss

14 Techniques to Accelerate Your Learning

19 мая, 2022 - 16:59
Published on May 19, 2022 1:59 PM GMT

Originally written by Belén Cobeta and Spencer Greenberg from ClearerThinking.org. Revised by Teis Rasmussen & Florence Hinder from ThoughtSaver.com. This is a cross-post from the Effective Altruism Forum.

TLDR: There are a number of techniques that may accelerate the speed at which you learn (14 of which we explain in this article). Additionally, embedded within this article are flashcards to provide you with the key takeaways so you can remember these ideas and more effectively put them into action.

Vasconcelos Library — photo by Diego Delso

What if you could learn more in less time? Whether you’re studying to pass your classes, aiming to improve at work, honing your personal life skills, or focusing on having the greatest impact you can, learning to learn more efficiently can be a great time investment because it accelerates the rest of your learning.

Many learning methods are inefficient

Many widespread learning practices waste a lot of time and effort, at least if we assume that the goal of those efforts is to actually learn. For example, you have probably had the experience of reading an exciting and useful piece of nonfiction, only to forget basically all of it and not take any action based on what you learned from it. And you’ve probably spent a lot of time taking classes that taught information that mostly wasn’t useful to you, and which you no longer remember either way.

It’s unfortunate that such experiences are as common as they are. The good news is that there are more efficient and powerful learning practices out there - many of us just don't adopt them. 

Here we lay out 14 of our favorite techniques for improving your learning processes. They are grouped into four categories:

  1. Learn faster
  2. Boost understanding
  3. Remember more
  4. Put your learning into practice
Part 1: LEARN FASTER

This section will cover techniques that can help you absorb more information or knowledge per hour that you spend learning.

1. Listening instead of reading

Listening is the new reading: take advantage of technology by listening to books and articles. It may feel less studious than reading, but at least some research shows that you can learn and retain just the same. The intonation of the narrator can also help you understand the text. Use the Audible version of the book or text-to-speech software and set the reading speed to a level that feels challenging but still allows you to understand the content. In time, your listening speed may become faster than your reading speed, as you go from listening at 1x to 2x and beyond. Some people even claim that faster speeds can improve comprehension to a degree because they require greater focus and so can prevent the mind from wandering. Another advantage of audio over regular reading is that you can do other activities that don’t use your conscious mind at the same time, like taking out the trash, walking outdoors, washing the dishes, or taking a bath. 

Check out The Nonlinear Library for audio-versions of content from blogs such as the EA Forum, Alignment Forum, and LessWrong. 

2. Immersive reading

Why choose between listening and reading when you can do both? Try Emerson Spartz´s #1 speed reading hack: read a book using the printed and the audible version simultaneously (that is, read with your eyes WHILE you’re also listening). By engaging two senses at once, your focus and reading speed may increase. It takes some practice, so be sure to try this method for at least a few hours before deciding if it’s right for you (people who just try it for one hour are unlikely to see a benefit). To get the full benefit, you’ll also want to push the speed of the audio to the upper edge of what feels comfortable to you.

3. Recursive sampling

Use this technique to help you decide how to allocate your reading time. Instead of diving right into a book, start by reading an article about it or a review of it. Most published books have blog posts or articles about them, so this shouldn’t be too hard to do. If, based on the article or review you read, the book seems like it will be valuable, then read the first three chapters of the book. If it still feels valuable, go ahead and read the whole thing. Apply a similar method when choosing what scientific articles to read: start with the abstract if there is one. Read the conclusion or the last few paragraphs next before you read the whole article.

Here is the first set of flashcards to provide you with all the key takeaways so you can remember these ideas forever.

Part 2: BOOST UNDERSTANDING

Here, we cover techniques to help you spot connections between concepts and expand on your mental models of how things work.

4. Intuition flooding

We often think of learning as an analytic process, which involves thinking carefully about concepts in order to understand them. But there is another important kind of learning, which involves developing a deep, intuitive grasp of a topic. You can work on developing this kind of understanding by using a technique that we call “intuition flooding”. To practice intuition flooding, find a large number of examples of something you want to understand (say, 20 to 100) and look at each of them carefully, one by one, scrutinizing them, while jotting down any interesting observations you have or patterns you notice. Here are a few examples:

  • To understand painting better, spend two minutes looking at each of 100 paintings of many different styles, scrutinizing each one carefully
  • To understand app user experience, take a close look at different user experiences in 50 different popular apps
  • To better understand depression, read 50 accounts of depressed people talking about what their experience of depression is like
  • To understand why civilizations fail, read brief accounts of the collapses of 20 different failed civilizations

If you pay really close attention to each example and look at enough of them, patterns will likely jump out at you. What’s more, you’ll likely start to develop an intuitive grasp of the subject even if you can’t fully verbalize it.

5. Mind mapping

This technique helps you understand a topic by uncovering the relationships between the most important concepts. Simply draw a picture to represent the main idea you want to grasp. As new concepts are introduced, put each of them in an oval, and then connect them with different types of lines to represent different sorts of relationships between those ideas. For instance, you might use an arrow to mean “causes”, a regular black line to mean “correlated with”, a thick line to mean “is a part of”, or whatever other types of connections you think are useful to highlight. The process of identifying these connections leads you to understand the idea at a deeper level, while the combination of text and diagrams helps your brain scan, synthesize and digest the information. 

For example, if you were learning about the Surrealist art movement, your mind map could look something like this:

6. Tell a friend

One way to test your understanding of a topic you just learned about is to try to explain it. You could do this by explaining the idea to an interested friend, writing a blog post summarizing it, or by pretending you are teaching it to a 6th grader (this last one is sometimes known as the “Feynman Technique”). As you are doing this, you’ll likely not only end up with more clarity in your understanding, but you’ll identify important gaps in your understanding. Go back to the source material to fill in those gaps.

7. Spot the Core

When learning about a complex idea, often one of the biggest challenges is figuring out the most important parts of the idea. A nice social-media method to do this is what we call “Spot the Core.” For the concept of interest, ask yourself — what is the most important part of this idea that I can express in one tweet? Then attempt to represent the core of that idea in just 280 characters. This forces you to focus on what is truly essential about the idea, which can deepen your understanding. A follow-up exercise once you’ve done this is to ask yourself what you would add to your first tweet if you could only use an additional 280 characters (so, two tweets worth of material).

8. Triangulating Genius

Suppose you want to learn about a complex topic where experts disagree with each other. This is tricky because it’s hard to even know where to begin, or which experts to trust. For instance, take the topic of “the power of intuition”. A quick search will show you that there are well-reputed scholars that solidly argue both in favor and against using your intuition to make decisions. To Triangulate Genius, pick two to four brilliant experts with differing perspectives on the topic.

Using the example of intuition from above, you may choose Gary Klein (famous for his work on the effectiveness of expert intuition), Daniel Kahneman (famous for his work uncovering many biases in our intuition), Gerd Gigerenzer (who challenged some of Kahneman’s work), and Phil Tetlock (who examined the poor forecasting abilities of many experts).

Or, if you’re trying to learn about the world of investing, you might choose as your geniuses Warren Buffet (generally believed to be the greatest value investor of all time), Ray Dalio (who started what was the world’s largest hedge fund), and Nassim Taleb (a famous contrarian in finance).

Once you’ve selected your experts, go ahead and read their arguments on the topic. Not only will you learn a lot from their ideas, but you can learn even more by taking note of what they agree and disagree on. The intersection points where these experts agree are the conclusions you can be most confident in. You should have less confidence in the points that they disagree on, and you’ll either have to defer judgment or form your own opinion based on the arguments each side makes.

9. Expert Observation

It is quite unlikely that you can master chess just by reading books and following tutorials. Although those are a good starting point, your game will most likely benefit from “expert observation” — watching experts play the game while they explain what they are doing and why.

Observational learning begins in childhood. We model the behavior of parents and teachers as well as friends and siblings. Expert Observation takes this idea to the next level by watching someone truly exceptional at a particular task. If you have the chance to ask questions during the process, that’s even better. Ideally, the expert carrying out their skill would explain what’s going through their head and why they make each decision the way they do. When the expert does something you don’t understand or that confuses you, that’s a great time to ask a question. These days, YouTube is a great place to look to find experts explaining their process as they carry out their craft.

Here is another set of flashcards to provide you with the key takeaways from the section “Boost Understanding.”

Part 3: REMEMBER MORE

In this part, we’ll cover techniques that can help you reduce waste by retaining more of the valuable information or knowledge that you’ve already acquired.

10. Active recall

Active recall is where you actively try to retrieve an answer from your memory before you look at the answer.

One famous study found that using active recall, students were able to remember 80% of what they learned long-term whereas, with traditional study techniques, students were only able to remember 36%.  

So how could you apply this? For example, if you were learning about space exploration, you could quiz yourself with a question like “How did the U.S. government justify the expense of going to the moon?”, and then when you review the content, take a moment to try to remember the answer (before revealing the answer), rather than merely reviewing the material passively. 

11. Spaced repetition

Newly acquired information quickly fades in memory when no effort is made to retain it, following a pattern known as the forgetting curve. After you have finished reading an article or a book chapter, some studies suggest that you will likely have forgotten more than half of it already within the hour! 

Image from the Farnham Street blog

Spaced repetition is a powerful way of counteracting this natural information decay. This method, based on the spacing effect, minimizes the time spent reviewing the learning materials while increasing long-term retention. It is most effective to schedule reviews of your learning materials at increasingly longer time intervals. This means that you should review what you just learned shortly after learning it (within a day or a week at most) and space out the reviews afterward: 

Image from the Farnham Street blog

If you noticed that you had trouble recalling the information, that would be a sign that you waited too long for the review, so schedule the next one within a shorter time interval. If you get the material right, then you can wait a longer amount of time for your next quiz.

To make this easier, you can give Thought Saver a try. It’s a free tool we made to accelerate your learning with active recall, spaced repetition, and our own approach to turning learning into action. It's designed to make it much easier to integrate spaced repetition into your life and to help you turn ideas into new, helpful actions.  

12. Incremental reading

This learning technique involves going through new learning material and reviewing old material at the same time, using a process that helps you understand difficult concepts that are new to you. Incremental reading is credited to Piotr Woźniak and is integrated in the newest versions of Supermemo, which he created. The method, as described below, is closer to Michael Nielsen’s take on incremental reading, which you may find a bit easier to learn and apply.

Suppose that you are reading a challenging set of articles on a topic that you are not familiar with, but which you are motivated to really understand. For instance, you may be trying to figure out how “deep neural networks” work.

Skim the articles once, making no attempt to fully understand them. The purpose at this stage is to identify the main ideas in the articles, notice what you already know and figure out the things that you need to learn about.

As you do this, take notes using advanced note-taking systems like Roam and Notion. When you encounter each important seeming concept and definition, turn them into flashcards using ThoughtSaver, Supermemo, or Anki, so that you can be quizzed on the content using the methods of spaced repetition and active recall. Review these cards regularly.

Make several passes through the articles this way (most likely spread over a few days or even a week), and you will notice that the flashcards you’re creating start going deeper into the topic. Once you’ve reviewed these more detailed flashcards a few times, you can now try reading the articles again, this time taking your time to try and understand them in detail. You will likely find that the concepts start to fall into place in your mind. What’s more, because of using spaced repetition, the content will likely stay in your memory much longer, and you have an easy way to brush up on the ideas at any time.

Here is another set of flashcards to provide you with the key takeaways from the section “Remember More.”

Part 4: PUT YOUR LEARNING INTO PRACTICE

It can be really difficult to turn knowledge and insight into valuable changes in behavior. It’s unfortunate that this aspect of learning is so often neglected, as it tends to require conscious effort. And the things you learn are only useful if they result in different outcomes. So in this final section, we’ll explore three techniques to help you take action based on what you learn.

13. Trigger-action plans (TAPs)

One way to turn learning into practice is by associating an idea or mental model with a situation in which you would like to apply it. This can be done by forming mental habits with the structure "When situation X arises, I will perform response Y". Put differently, “When I encounter this trigger, I will perform that action”. This is what the psychological research literature calls “implementation intentions”. The Center For Applied Rationality (CFAR) instead refers to the catchier “trigger-action plans” (TAPs). 

The triggering situation could be just about anything; it could be something external, such as your immediate physical environment. Or it might be something internal, like a certain emotional state. For example, you might want to associate shallow breathing or racing thoughts (trigger) with the action of closing your eyes and taking a few deep breaths to calm yourself. Or when you enter your office or apartment building, you might want to simply look at the staircase (as opposed to the elevator) to get yourself to take the stairs and get a bit of exercise. 

The idea is to make goal-directed behavior automatic. Ideally, by mentally associating a trigger situation with a desired action, you will effortlessly take that action once the trigger is encountered. 

To create new trigger-action patterns, CFAR has recommended that you mentally rehearse it a total of ten times. Yes, ten times! That is, either think or say to yourself "When situation X arises, I will perform response Y" ten times. You can use ThoughtSaver to automatically remind yourself of the TAPs you’ve set. Or you can use our free Clearer Thinking tool “Program Yourself” to create new TAPs. 

14. Project-based learning (PBL)

A more hands-on approach to turning learning into practice is to work on a small project using the skills you are trying to learn. The advantage of PBL is that it increases your engagement with, and motivation towards learning. Additionally, it focuses your learning on skills and knowledge that you actually intend to use.  

For example, if you want to learn data science you can pick a project that you are particularly interested in, such as analyzing a data set about air pollution's impacts on health. You can look up the resources needed, run through some example questions on the Kaggle data science competition site, and then finally apply what you learn to a particular dataset that seems interesting to you. You make something of your own by the end of the process, creating a sense of personal achievement and satisfaction.

At the same time, by using PBL you can often create tight feedback loops where you quickly get high-quality feedback on your performance (as you see where you are making progress and where you are stuck), which can dramatically speed up your learning. In addition, it helps you understand the subject matter more deeply when you put it into practice; you’re naturally encouraged to notice gaps in your understanding, as you often won’t be able to do what you’re trying to do without filling those gaps. 

For example, you probably won’t be able to do data science without knowing some programming. If you were just doing an online course, the temptation to simply skip content related to improving your programming skills might be too high, as the consequences of doing so might feel abstract and distant in time when you’re not putting it into practice right away. 

Finally, Project-based Learning can be useful as a way of combining a favorite hobby or passion with learning a desired skill set, making it more fun and engaging to learn something new. For example, if you love graphic design, you could design an infographic to complement your data analysis.

You’ve just seen fourteen techniques for accelerating your own learning. We hope that you’ll take a moment right now to pick one or two of them to apply right away. You can start learning faster today! 

We also would encourage you to check out our app ThoughtSaver.com to help you to continue applying these concepts. You can also click below to get a flashcard deck we made to help you remember the fourteen learning techniques covered in this article.

Here is our last set of flashcards to provide you with the key takeaways from the section “Project-based Learning.”



Discuss

Covid 5/19/22: The Law of Five

19 мая, 2022 - 15:20
Published on May 19, 2022 12:20 PM GMT

If you’re five, you now get a booster. If you’re four, you still get no vaccine at all (while we ‘wait for more data’) and you’re often forced to take precautions that might make sense for senior citizens.

Because no one cares. And that truly continues to be the overall message. No one cares about the physical prevention or mitigation of Covid. Congress allocates zero dollars, doctors don’t prescribe Paxlovid and life mostly continues as normal because Omicron in a mostly-already-infected-and/or-vaccinated population simply isn’t that dangerous. Cases go up, but life goes on.

It’s pretty terrible that we’re not funding pandemic prevention or anything for the current pandemic, but at this point I don’t know that the most likely alternatives are even improvements.

(And of course, if you’re under one, thanks in large part to the FDA you may have also have a problem getting food.)

Florida did not report this week. I am retroactively smoothing out Florida’s numbers by giving 50% of its reporting weeks to its no-reporting weeks, now that this has happened three times in a six-week period. All future numbers and predictions will include this adjustment (assuming I remember to do it) and I will assume up front that when Florida reports we take 50% off the top. If I find time I’ll do a smarter adjustment, since this one is slightly flawed, but it’s much better than doing nothing.

Executive Summary
  1. Cases up.
  2. Boosters available for ages 5-11.
  3. No one cares and deaths are not rising yet.

Let’s run the numbers.

The Numbers Predictions

Prediction from last week: 640,000 cases (+23%) and 2,100 deaths (+1%)

Results: 600,742 cases (+23% after adjusting Florida for both weeks, would have been +15% without the adjustment) and 2,004 deaths (+2% after Florida adjustment, -3% without the adjustment).

Prediction for next week:

Deaths

Deaths are rising in the Northeast and flat elsewhere, despite it being more than three weeks since cases began rising again. It seems clear that case fatality rates in this wave are going to be lower. It’s not great that thousands of Americans with Covid are dying each week, some of them from Covid as well. I wish it were not so, but I am glad we are not ruining our lives trying to stop this.

Cases

Enough weeks in a row with 20% weekly case growth and things get out of hand. We’re now triple our low point and back at a point similar to where we’ve been for most of the past year. I expect case growth to peak well before we get back to where things were in January, and even if we did get those case numbers back I expect less death to result from it, but certainly this is not where I hoped or expected to be in May.

The rise of BA.2.1.12 (and BA.4 and BA.5) are something I wasn’t predicting would happen, or at least would not happen so fast. Variants are inevitable, but this was unexpectedly quick, and Topol’s note below that this implies more future variants is well taken.

Is the pandemic over? In terms of cases, we have a clear no. In terms of deaths and people’s reactions, that remains to be seen. I still consider my pandemic over.

Physical World Modeling

Eric Topol describes our current approach as The Covid Capitulation. That name seems apt, for both good aspects and bad. The first half of the post is about how B.2.1.12 and B.4/5 bring both the new mutations and doom.

To recap, we have a highly unfavorable picture of: (1) accelerated evolution of the virus; (2) increased immune escape of new variants; (2) progressively higher transmissibility and infectiousness; (4) substantially less protection from transmission by vaccines and boosters; (5) some reduction on vaccine/booster protection against hospitalization and death; (6) high vulnerability from infection-acquired immunity only; and (7) likelihood of more noxious new variants in the months ahead

This is repeating ‘the virus mutates a lot’ over and over. It’s worth repeating, as it is central to modeling what will happen in the future. As one would expect, static vaccines and existing immunity decline in effectiveness as we accumulate mutations, and immunity from both infection and vaccination are now established as fading at least somewhat over time especially for purposes of initial infection.

The flip side of this is that there have been a massive number of infections, and there will continue to be a lot of them, without many deaths. This creates a cycle where immunity is renewed was time passes and variants give way to new variants and subvariants.

This graph is worth copying over, showing a clear decline in vaccine effectiveness and booster effectiveness. It is still showing vaccine and booster effectiveness, and all the usual biases in the samples continue to apply, but the ratios are no longer anything like as dramatic as they used to be.

One reason for this is that previously almost all infections were first-time infections. Now many of them, perhaps a majority, are re-infections – by the math, they have to be, there have been roughly as many infections as we have people. A lot more of the unvaccinated group is going to have been previously infected, and especially been previously infected multiple times, and those infections will ‘buy more protection’ for them than they will for the vaccinated group. So this results in less of an advantage for the vaccinated even if underlying effects of vaccination hadn’t changed, and this is doubtless an important part of the story.

Topol comes out in favor of a policy of Zero Covid Deaths, as opposed to Zero Covid. The problem is that they’re the same policy. You can’t have a virus running around infecting lots of people and killing zero of them, no matter how good your vaccines and treatments. That’s not how the physical world works. So such slogans are always calls to Do More. But the core idea of shifting to mitigation of the death rate is clearly correct at this point.

The second half of the post is called A Delusional Congress and is about congress deciding to fund exactly nothing towards either the current pandemic situation, work to prepare for future variants, or future pandemic prevention.

In the opposite of news, I strongly agree with this central call.

We’re not just looking at running out of vaccines and antiviral medications. Congress should immediately allocate for an Operation Warp Speed (OWS)-like initiative to bring nasal vaccines over the goal line. Three of these are in late stage clinical trials and success of any would markedly ameliorate our problems of transmission, no less the alluring aspect of achieving mucosal immunity and being variant-proof. That brings us to catalyzing the efforts for a pan-β-coronavirus vaccine, previously reviewed, now that we have discovered tens of broad neutralizing antibodies but have limited traction of these in the form of advanced clinical trials.

What I disagree with is Topol’s characterization of why congress is failing us. Congress isn’t unusually delusional. They know that ‘what they are doing is wrong’ in the relevant sense. They just don’t… care. Not enough to act.

Did you know that the new version of the Moderna vaccine is more effective?

The booster candidate, mRNA 1273.211, was tested n 895 people against the currently approved mRNA 1273 booster. Some participants received a 50 micrograms (µg) dose, the same amount of viral RNA as the current boosters. Some received a 100 µg dose.

The participants who received the 50-µg dose had a 2.20-fold increase in their neutralizing antibody titers against Omicron after 28 days. Six months later, the participants still had 2.15 times the antibodies acquired with the original Moderna booster.

That’s not that great. Doubling your antibodies is a lot less than doubled effectiveness, as all of this needs to be thought about on a log scale. It’s still something we should be willing to spend a little money on to ensure it happens quickly and also at all.

The battle against Paxlovid continues. The most effective weapon is pure ignorance, followed closely by a race between confusion and an obsession with conserving supply and whether someone ‘needs’ it, since Covid isn’t all that dangerous for most people, you see. On the drug interaction front, Mina points us to good resources. Drug interactions are a worry, but they’re still relatively rare, applying to 2 of the 100 most prescribed drugs. The vast majority of people won’t have an issue.

Boost Your Children Well

If you are five years old, you can now get a Covid booster. Washington Post coverage is here.

If you are four years old, you can’t get the vaccine at all, often with maddening social consequences.

I do not notice that any particular urgency to get my eight year old son a booster.

Also, I continue to be impressed with the flexible definitions of the word ‘emergency.’

The FDA also announced plans to meet June 7 with its advisers to talk about an emergency authorization request for a coronavirus vaccine made by Novavax for people 18 and older.

Novavax submitted an application for emergency use authorization in late January.

It would be good to approve Novavax on principle. In practice I doubt anything changes. It’s great to know your emergency application will be handled only five months later.

There’s also this:

In addition, the FDA said it will convene an advisory committee meeting June 28 to discuss whether the composition of coronavirus vaccines should be modified for use in a potential vaccination campaign this fall.

Many experts say vaccines for the general population should be reformulated to keep up with the changing virus, but the question of whether, when and how remains unresolved.

So over a month from now they will convene to discuss whether to reformulate vaccines going forward. Presumably, this allows the FDA to say that there’s no hurry because congress isn’t funding further vaccinations, and the congress to say there’s no hurry because the FDA hasn’t approved updating the virus.

I remember when we said mRNA vaccines could be modified in a week.

I also remember when we designed them in a day.

But the CDC is worried. They’re worried that if we have an effective vaccine program, it might give the impression we don’t have an effective vaccine program?

If I wanted to give people the impression I had an effective vaccine program and that vaccination was important, I would act like vaccination was important and do my best to have an effective vaccine program.

Every single time the Very Serious People sabotage our pandemic response in the name of ‘impressions’ people correctly get the impression that we are sabotaging our pandemic response. It backfires. Every. Single. Damn. Time.

That doesn’t mean you want to do a big push to tell everyone an additional booster is super important. The argument for ‘saving your powder’ on that front makes sense.

Prevention and Prevention Prevention Prevention

More free Covid tests are available. The last of our Covid money is being spent on this.

Covid-related entry requirements have been waived by many European countries, do nothing, waste a lot of time and money, are dumb, and we should scrap them.

New York City issues new mask advisory.

This continues to be brutal on young children, who are at almost zero risk yet are told to ‘avoid non-essential get-togethers’ also known as childhood, because they are not eligible for vaccination. My son still has to wear a mask each day. There’s also the continued baseline assumption of cloth masks rather than N95s, and no mention of masks better than N95s.

FDA Delenda Est has rejected the emergency use application for Fluvoxamine. They did so by looking around until they found a set of endpoints that had insufficiently strong results, and deciding those were the ones they cared about.

I didn’t expect better on Fluvoxamine, because all signs point to no one caring about Covid. It’s not enough of an emergency to let me (yet) vaccinate my kid, so why should I expect this decision to go any differently? Plus the FDA thinks that effects that aren’t sufficiently dramatic should be illegal – a ~35% reduction in severe disease seems a lot to them like a 65% chance of them getting blamed, and a lot of confusion they don’t want to deal with. Much better, they think, if we all pretend the whole thing has gone away.

What both these decisions have in common is that those who know better can ignore them. Fluvoxamine is already approved for psychiatric use, so you can still get a prescription if you care enough and work the system. An advisory is a rule you don’t have to follow. The real damage is from rules you have to follow.

The Bob Wachter Chronicles: Paxlovid Postscript

Last week Bob’s wife got Covid and had to decide what to do about it. This week, she took Paxlovid and she’s feeling better, but worrying about the situation is far from over. Another thread into the mind of the well-meaning, well-informed and overthinking Covid concerned – if you’re not here for it, skip ahead to the next section.

The Long Covid paranoia always makes my heart sink. In this case, it’s concern after getting Covid rather than before, so it will stress everyone involved out but at least won’t prevent them from living life. This was a (mild) symptomatic case so there’s certainly a non-zero chance.

Then there’s the concern about Paxlovid rebound. This is the New Hotness concern. I see a decent number of tweets that have one particular case of it and say ‘not rare.’ That is… not how statistics works. There is some motivation for people to play this up, as part of the general motivation among health professionals to play down Paxlovid.

It’s worth noticing that the survey results above both don’t allow for answers below 2% despite 2% being the rate at which the effect has previously been reported, and also looks exactly like it would if people had actual no idea what the answer was.

What’s Bob’s answer?

Excuse me while I bang my head against the wall.

There’s the whole ‘we don’t know for sure so we have no idea’ thing but also:

A similar percentage was seen in the placebo group.

Technically it was 2% in the treatment group, 1.5% in the placebo group.

You know what this suggests? That we should be skeptical that most of the rebounds we observe have anything to do with Paxlovid. Yes, we have a reasonably good ‘just so’ story about them such that it makes sense that they happen. But what Bob is reporting is that the rebound effect was not associated with Paxlovid in the trial, at all. One could reasonably thus say both that Paxlovid is unlikely to cause that many rebounds, and that if you want to take this seriously than you should be worried about ‘rebounds’ in all Covid cases even without Paxlovid. Which we are totally not.

The good rebuttal to this that I have seen is that Omicron gives you symptoms faster, so Paxlovid is getting prescribed earlier, so we end up giving out Paxlovid ‘too quickly’ and thus rebounds are happening more often. That might even be an argument that you want to wait to take Paxlovid if you think you caught your case super quickly. In this case, Bob’s wife didn’t get started for about two days after detection, which means this likely isn’t a concern.

Bob’s right on the question in #12 here. If your test comes back positive, that implies you’re infectious and you should act accordingly.

However, in the question in#14, we’re not talking about what Bob should do if his wife tests positive again. We’re talking about what he should do given his wife is still testing negative, except that she previously got Covid, took Paxlovid on the 8th and tested negative three days ago. There’s no sign of a rebound case, and without a rebound she’s in the clear.

(Also note, they did a nine hour car ride together when she was definitely positive, using only N95s, but consistency is mostly not a reasonable ask.)

If our only prior is ‘got Covid and then took Paxlovid’ we could reasonably put his wife at a few percent to have a mild rebound case at some point, which might or might not make her infectious at this point, and which is reduced by all the days of negative testing so far. It’s also reduced by the details of her case. She had several days of symptoms, which makes the physical logic of a rebound seem much less likely. Thus I voted for option one, but I certainly won’t blame anyone who wants to open a window.

As you’d expect Bob chose to wear masks for an additional week. My guess is that given his emotional reaction in all scenarios, it was the right choice for him. If you’re ‘being careful’ this is part of the storytelling package of being careful.

(As a side note, everyone tells me Better Call Saul is terrific, and objectively that seems very right, but I noticed I wasn’t enjoying watching it so I stopped after a few episodes. I presume I should eventually push through that.)

My answer to the question of another round is no, because from what I know the rebound cases aren’t as serious, so there’s no need.

I consider the ‘not an FDA approved use’ justification here to be rather content-free. Yes, of course there haven’t been enough rebound cases for us to know if the second course is wise here, but we should be able to use our physical world models to figure that problem out.

On the last question, 3% is basically Lizardman Constant, so it doesn’t worry me. The 19% part does, yet it also raises a valid point. If using Paxlovid means several weeks of living as if you are positive, that’s a non-trivial downside. If that’s how you’re going to react, the calculus does not look as good. It still looks pretty great, though, if you’re in a medium-risk or high-risk group.

Here’s a link to that post. Her takeaway seems to be that yes we are seeing a lot of anecdotes, but that’s not inconsistent with a 2% rebound rate, and we have to wait for better data which will be hard to get.

The New Normal

Hotel occupancy is back to pre-pandemic levels, but NYC subways very much are not.

About half of subway ridership has vanished and so far it isn’t coming back. I buy that this is a shift towards massively more working from home and that the bulk of this change likely never reverses.

Whereas hotels represent activity that isn’t going away. Although there’s another explanation here, which is: Hotels, man. Capacity can be down, as can prices and profitability, and occupancy being at equilibrium may not mean much.

I asked someone who knows the business and he said inventory is down generally, so this is a ‘new normal’ of sorts but does not represent anything like a full rebound. But at the new supply levels prices look more like they used to look, whereas when I visited New York during the pandemic hotels were remarkably cheap.

In Other News

Well-earned congratulations are in order: Zeynep Tufekci accepts position at Columbia University as inaugural director of Center for Ethics and Security in Journalism. Turns out it was always about ethics in journalism.

New York Times publishes helpful front page population density map of America.

Meat packing plants may have been a remarkably large share of early Covid infections.

Marriage rates declined 16% in 2020. I was very surprised that this effect was so small.

Ministry of Truth Proves False

Ministry of Truth closes its doors in wake of the sharing of information about the Ministry of Truth, the idea of creating a Ministry of Truth and its new would-be leader who has a history of calling for censorship of any speech she doesn’t like. The framing by the Washington Post was that this was an attack by exactly the kind of misinformation the board was to be deployed against.

Which tells you exactly what kind of misinformation the Ministry of Truth was designed to censor.

That the would be 33-year-old leader who has gone around calling for censorship is called a ‘well-respected expert’ in the field of misinformation tells you what it means to be a ‘well-respected expert’ these days.

Pirate Wires has more details and less generosity to offer, if that’s relevant to your interests.

I would like to better understand the mechanisms whereby this backlash resulted in the board shutting down. None of the reactions seem like they should have been unexpected, or like they should have any unexpected impacts. I notice I am (pleasantly) confused that they worked.



Discuss

What does failure look like?

19 мая, 2022 - 10:59
Published on May 19, 2022 12:59 AM GMT

Not related to AI.

I'm quite worried at the moment that there's some way that I can fail, become depressed, reclusive, and not achieve goals that's subtle enough that I won't notice any mistakes I'm making. I'm quite confident I won't become an alcoholic and won't take hard drugs, but there might be so many other ways to fail that are slower and harder to spot.

 

What are these ways? How do I avoid them? 



Discuss

Should you kiss it better?

19 мая, 2022 - 06:58
Published on May 19, 2022 3:58 AM GMT

Whenever my toddler hurts himself, I offer to kiss it better. This works like magic - he immediately stops crying and carries on playing happily. Placebos are wonderful things.

Doing this makes me happier, it makes him happier, and also serves as a useful diagnostic tool - if he carries on crying I know he's actually hurt himself badly.

Teaching children that kisses can make things better is definitely lying to them, but this seems like a good place to make a principled exception. WDYT?



Discuss

A possible check against motivated reasoning using elicit.org

19 мая, 2022 - 02:28
Published on May 18, 2022 8:52 PM GMT

Are you worried you may be engaging in motivated reasoning, rationalization ... or committing other reasoning fallacies?

I propose  the following epistemic check using Elicit.org's "reason from one claim to another" tool

Whenever you have a theory that A→B.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , 
Take your  theory negating one side or the other (or the contrapositive of either negation), and feed it into this tool.

Feed it
A→¬B

 and/or 
¬A→B,

and see if any of the arguments it presents seem equally plausible to your arguments for A→B.

If they seem similarly plausive, believe your original arguments and conclusion less. 

Caveat: the tool is not working great yet, and often requires a few rounds of iteration, selecting the better arguments and tell.ing it "show me more like this", or feeding it some arguments.

When in Rome ... do or don't do as the Romans do?

Discuss

Feminism and Femininity

19 мая, 2022 - 01:53
Published on May 18, 2022 10:53 PM GMT

Is it feminist to wear lipstick? There are three possible views on the position that feminist thought should take with respect to expressions of conventional femininity in patriarchal societies: 

  • Expressing femininity is actively good
  • Expressing femininity is actively bad
  • Women should do whatever makes them feel most comfortable
  • Femininity is a distraction - masculinity is the problem  

These different views express a combination of different views on the mechanism by which patriarchy operates, the most effective way to advance women's liberation, and the duties which women have to their gender. 

How does patriarchy work

There are three competing views of how patriarchy might work. Patriarchy could be a result of gender socialisation. This theory goes that the way in which gender is socialised holds women back from holding power, achieving status and having good life chances - or some combination of all three. In contemporary Western Europe this mostly manifests as relatively soft norms where, for instance, women are more harshly punished for expressing anger. Probably the most important substantiation of gender socialisation as preventing women from achieving power and status - although not necessarily good life outcomes - is the norm where women sacrifice their careers when raising children with a male partner. In less gender equal societies this can take on a much more extreme form, where the common pattern is the barring of women from the public sphere. This describes societies such as classical Athens or 19th century Pashtun farmers where women are excluded from commerce and politics - the things that give men power.

The competing view is that patriarchy is defined by a value system rather than a set of norms. In the former theory what was deemed to be high status things - the axis along which inequality should be measured - was taken as a given. It was participating in the public square. Under this other view however, patriarchy comes from the work women do and femininity per se being defined as less valuable than the work men do and masculinity per se. 

There’s a strong version and a weak version of this claim. The strong version says that it really would be fine if women stayed within the private sphere if that work was valued equally to the work in the public sphere. To make this concrete, and staying within the context of capitalist societies in Western Europe, we could pay women equal amounts to the average male salary for care work that women disproportionately provide, both as a way of concretely giving women more power and better lives, but also because money is a key way in which these societies express who has, and what roles, are high status. This could be extended to other forms of traditionally female work in the public sphere, such as nursing and cleaning. 

However, there’s also a weaker and subtly different form of this claim. This version doesn’t argue that women should keep to their traditional roles but that these roles should be higher status, it argues that women should keep to traditional versions of femininity and society should change to value these traits more highly. For instance, rather than women adopting the risk taking attitudes of men, society should change to value lower risk more highly. 

The final view is that it’s nothing women do that’s the problem - it’s male power and often specifically male violence. This is the logic behind the focus on male violence against women despite the fact that much are much more at risk from homicide. In this worldview, male violence against women isn’t simply a tragedy it’s a part of how the patriarchy controls women. Murder is only the most extreme form of this. Domestic violence and sexual violence are probably the next rungs down the ladder, followed by sexual harassment and right down to, for instance, interrupting women. There’s also a more specific phenomena of women who don’t follow gender norms in some respect being harmed by men because of this. A good example of this is instance Margret Thatcher being portrayed as a man on spitting image, the joke being that she was domineering like a man so lets portray her as a man to take the piss haha very funny. Outside of a contemporary Western European context, honour killings are an extreme substantiation of this. 

Should feminists wear lipstick?

These three theories of patriarchy have very different implications for whether feminists should wear lipstick - i.e the degree to which women should express their femininity and much more generally how we should structure policy. The first view - where women are socialised to not challenge men’s position in the public space - implies that we should aim for a breaking down of femininity and gender and more generally. Gender itself is the rope that binds women. To put this in more concrete terms, this would imply that, for instance, we should be pushing back hard against little girls wearing pink and going to dance classes. For a more policy relevant example we should be extremely heavily incentivising men to take paternity leave with the goal of bringing men as equally into the private sphere as women so that women can take their fair share of the public sphere. 

If one believes the second then most definitely feminists should wear lipstick but not only that, women not expressing femininity in the public sphere is as anti-feminist as the sexualization of female politicians is under the first theory. By not expressing femininity in the public sphere women enforce the notion that femininity is inferior to masculine traits and makes it harder for other women to follow them. The strong version of this theory takes this argument even further. By privileging the public sphere over the private we’re implicitly subscribing to male values. This would imply that we should be paying mothers and other caring roles that women disproportionately take on as we would pay any high status, important job. 

Finally, the third theory says that the question of whether feminists should wear lipstick missies the point at best and a worst is another way of putting the onus on women to change their behaviour around the fixed point of men. The extreme version of this position argues for curfews on men to allow women to be able to go out in public free from the spectre of male violence. Less extreme ones argue for a shift in policing priorities to focus more male violence against women, harsh penalties for sexual harassment and strong norms against things like men interrupting women (and hopefully not for men writing articles about feminism who haven’t even read anything by Judith Butler.) 

Politics is about action

It’s not enough to merely know which one of these theories best describes how patriarchy works - politics is about action and the best action to take is determined by the constraints you face and your normative judgements. 

I think one of the key issues in deciding what policy course to take is the degree to which gender - and I mean gender not sex - is biologically determined rather than socially constructed. If one believes that it’s likely that women will always have a much stronger urge to spend a substantial portion of their lives looking after their children then it becomes a much higher priority to raise the status and pay of child rearing and care work more generally. To emphasise the point I’ve made a number of times I believe that there are trade offs here. If one believes that gender socialisation is the key culprit then raising the status of women spending a large portion of their lives as caregivers reduces the incentive for women to enter the public sphere and the traditionally male role that that entails and so is actively harmful to women's liberation. 

A second key consideration is how much one believes that what we value now is good. If it genuinely is the case that, for instance, being very willing to take big risks is very socially valuable this should push one towards attempting to change what femininity entails rather than attempting to devalue risk taking. Conversely, one could believe that what we currently value - both intrinsically and in terms of valuable traits - is deeply harmful. In this case, in addition to it being good for gender equality, having society value roles currently associated with femininity will make all of society better off. For instance, working extremely long hours at the expense of your family might be worth it if it’s the only way to create the most innovative products. Alternatively, if people are mostly working very long hours to create better versions of flavoured sugar water, maybe it would be good if society valued spending time with one's family more highly.

The question of how patriarchy arise and perpetuate is of course an extraordinarily complex question that requires careful empirical work and it’s beyond the scope of this post to review that literature here. 

The other core question of politics is normative - what is the right thing to do. I think it’s this lens that justifies the view that women should simply do what makes them most comfortable. This post has been written from an implicitly feminist perspective. The core normative feature of this perspective is that individuals have a special responsibility to reduce gender inequality - in some sense women who don’t are traitors to their gender. In the language of moral philosophy everyone and potentially women in particular have a duty to fight against patriarchy and an action which has gender at its core is only permissible if it reduces gender inequality. The view that women should do what makes them most comfortable is fundamentally a liberal position. Individuals don’t have special responsibilities to their identity groups even if those identity groups are oppressed. It may very well be a praiseworthy thing to reduce gender inequality in the way one lives one's everyday life, but it is not blameworthy if one does not, especially if one is prioritising other morally important goals. Think the busy doctor who focuses on saving the lives of her patients and takes the path of least resistance with respect to other issues. 

Synthesis 

Throughout this post I’ve emphasised the need for tradeoffs. But ideally one would practise policies that improve the lives of women regardless of the theory of patriarchy that one subscribes to. This might push one towards policies aimed at reducing domestic violence in high income countries. In low income countries, where because the cost of living is so much lower one can often have vastly more impact with the same amount of money, this might imply improved access to financial services to allow women to be more independent of their male relatives. Politics is about action and effectiveness. This post will have been a success if it, very marginally, helps us take better action.



Discuss

What's up with the recent monkeypox cases?

19 мая, 2022 - 01:06
Published on May 18, 2022 10:06 PM GMT

Is it a thing I should be allocating attention to?

Wikipedia tells me that human-to-human transmission of monkeypox is pretty rare/difficult. There has been some community spread recently at least in the UK, but that's been speculated to have been from sex. On the other hand, there's been a bunch of cases reported in Portugal and Spain recently as well. Is that just normal background rate being given more attention than usual by the news cycle? Or is this actually an unusually high number of cases? Most importantly, is the number of cases significant evidence of increased human-to-human transmission?



Discuss

We have achieved Noob Gains in AI

19 мая, 2022 - 00:25
Published on May 18, 2022 8:56 PM GMT

TL;DR I explain why I think AI research has been slowing down, not speeding up, in the past few years.

How have your expectations for the future of AI research changed in the past three years? Based on recent posts in this forum, it seems that results in text generation, protein folding, image synthesis, and other fields have accomplished feats beyond what was thought possible. From a bird's eye view, it seems as though the breakneck pace of AI research is already accelerating exponentially, which would make the safe bet on AI timelines quite short.

This way of thinking misses the reality on the front lines of AI research. Innovation is stalling beyond just throwing more computation at the problem, and the forces that made scaling computation cheaper or more effective are slowing. The past three years of AI results have been dominated by wealthy companies throwing very large models at novel problems. While this expands the economic impact of AI, it does not accelerate AI development.

To figure out whether AI development is actually accelerating, we need to answer a few key questions:

  1. What has changed in AI in the past three years?
  2. Why has it changed, and what factors have allowed that change?
  3. How have those underlying factors changed in the past three years?

By answering these fundamental questions, we can get a better understanding of how we should expect AI research to develop over the near future. And maybe along the way, you'll learn something about lifting weights too. We shall see.

What has changed in AI research in the past three years?

Gigantic models have achieved spectacular results on a large variety of tasks.

How large is the variety of tasks? In terms of domain area, quite varied. Advances have been made in major hard science problems like protein synthesis, imaginative tasks like creating images from descriptions, and playing complex games like Starcraft.

How large is the variety of models used? While each model features many domain specific model components and training components, the core of each of these models is a giant transformer trained with a variant of gradient descent, usually ADAM.

How large are these models? That depends. DALLE2 and AlphaFold are O(10GB), AlphaStar is O(1GB), and the current state of the art few shot NLP models (Chinchilla) are O(100GB).

One of the most consistent findings of the past decade of AI research is that larger models trained with more data get better results, especially transformers. If all of these models are built on top of the same underlying architecture, why is there so much variation in size?

Think of training models like lifting weights. What limits your ability to lift heavy weights?

  • Data availability: (Nutrition) If you don't eat enough food, you'll never gain muscle! Data is the food that makes models learn, and the more "muscle" you want the more "food" you need. When looking for text on the internet, it is easy to get terabytes of data to train a model. This is harder for other tasks
  • Cost (exhaustion): No matter how rich your corporation is, training a model is expensive. Each polished model you see comes after a lot of experimentation and trials, which uses a lot of computational resources. AI labs are notorious cost sinks. The talent they acquire is expensive, and in addition to their salaries the talent demands access to top of the line computational resources.
  • Training methodology (What exercises you do). NLP models only require to train one big transformer. More complex models like DALLE-2 and AlphaFold have many subcomponents optimized for their use cases. Training an NLP model is like deadlifting a loaded barbell and training AlphaFold is like lifting a box filled with stuff: at equivalent weight, the barbell is much easier to pick up because the load is balanced, uniform, and in one motion. When picking up the box, the weight is unevenly distributed which makes the task harder. Alphastar was trained by creating a league of AlphaStars which competed against each other in actual games. To continue our weightlifting analogy, this is like a higher rep range with lower weight.

Looked at this way, what has changed over the past three years? In short, we have discovered how to adapt a training method/exercise (the transformer) to a variety of use cases. This exercise allows us to engage our big muscles (scalable hardware and software optimized for transformers). Sure, some of these applications are more efficient than others, but overall they are way more efficient than what they were competing against. We have used this change in paradigm to "lift more weight", increasing the size and training cost of our model to achieve more impressive results.

(Think about how AlphaFold2 and Dalle-2, despite mostly being larger versions of their predecessors, drew more attention than their predecessors ever did. The prior work in the field paved the way by figuring out how to use transformers to solve these problems, and the attention comes from when they scaled the solution to achieve eye popping results. In our weightlifting analogy, we are learning a variation of an exercise. The hard part is learning the form that allows you to leverage the same muscles, but the impressive looking part is adding a lot of weight.)

Why and how has it changed?

In other words: why are we only training gigantic models and getting impressive results now?

There are many reasons for this, but the most important one is that no one had the infrastructure to train models of this size efficiently before.

Hardware advances

The modern Deep Learning / AI craze started in 2012, when a neural network called AlexNet won the imagenet challenge. The architecture used a convolutional neural network, a method that had been invented 25 years prior and had been deemed too impractical to use. What changed?

The short answer? GPUs happened. Modern graphics applications had made specialized hardware for linear algebra cheap and commercially available. Chips had gotten almost a thousand times faster during that period, following Moore's law. When combined with myriad other computing advances in areas such as memory and network interfaces, it might not be a stretch to say that the GPU which AlexNet ran on was a million times better for Convnets than what had been available when Convnets were invented.

As the craze took off, NVIDIA stared optimizing their GPUs more and more for deep learning workloads across the stack. GPUs were given more memory to hold larger models and more intermediate computations, faster interconnects to leverage multiple gpus at once, and more optimized primitives through CUDA and cuDNN. This enabled much larger models, but by itself would not have allowed for the absolutely giant models we see now.

Software advances

In the old days of deep learning, linear algebra operations had to be done by hand... well not really, but programming was a pain and the resulting programs used resources poorly. Switching to a GPU was a nightmare, and if trying to use multiple GPUs would make a pope question their faith in god. Then along came Caffe, then tensorflow, then pytorch, and suddenly training was so easy that any moron with an internet connection (me!) can use deep learning without understanding any of the math, hardware, or programming that needs to happen.

These days, training an ML model doesn't even require coding knowledge. If you do code, your code probably works on CPUs or GPUs, locally or on AWS/Azure/Google Cloud, and with one GPU or 32 across four different machines.

Furthermore, modern ML platforms will do under the hood optimization to accelerate model execution. Easy to write ML models now are easy to share, readable, well optimized, and can be scaled easily.

Research innovations

There are two sets of important advances that enabled large scale research. The first are a legion of improvements to methods that allowed for less computational to achieve more. The second are methods that allow for more computation to be thrown at the same problem for better and faster results. While there are also many advances in this field, three stick out: transformers, pipeline parallelism, and self supervised learning

Transformers are models that run really well on GPUs by leveraging very efficient matrix multiplication primitives. They were originally designed for text data, but it turns out that for models whose size is measured in gigabytes, transformers are just better than their competition for the same amount of computation.

If we think back to our weightlifting analogy, transformers are like your leg muscles. For sufficiently large loads, they can't be beat!

(As an ML Systems researcher, every new application where just throwing a big transformer at a problem beats years of custom machine learning approaches and panels of experts brings me a strange joy.)

Pipeline parallelism is a bit complicated to explain to a nontechnical audience, but the short version is that training a machine learning model requires much more memory on the GPU than the size of the model. For small models, splitting the data between GPUs is the best approach. For large models, splitting the model across GPUs is better. Pipeline parallelism is a much better way of splitting the model than prior approaches, especially for models which are larger than a gigabyte.

Pipeline parallelism is like having good form. For smaller lifts its not a big deal, but it is critical for larger lifts.

Self supervised learning is like making flashcards to study for a test. Ideally, your teacher would make you a comprehensive set of practice questions, but that takes a lot of effort for the teacher. A self directed student could take data that doesn't have "questions" (labels) and make up your own questions to learn the material. For example, a model trying to learn English could take a sentence, hide a bunch of words, and try to guess them. This is much cheaper than having a human "make questions".

Self supervised learning is like cooking your own food instead of hiring a personal chef for your nutritional needs. It might be worse, but it is so much cheaper, and for the same price you can make a lot more food!

How have those underlying factors changed in the past three years?

TL;DR not much. We haven't gotten stronger in the past four years, just did a bunch of different exercises which used the same muscles.

All of the advances I mentioned in the last section were from 2018 or earlier.

(For the purists, Self supervised learning went mainstream for vision in 2020 by finally outperforming supervised learning).

Chips are not getting twice as fast every two years like they used to (Moore's law is dying). The cost of a single training run for the largest ML models is on the order of ten million dollars. Adding more GPUs and more computation is pushing against the amount that companies are willing to burn on services that don't generate money for the company. Unlike the prior four years, we cannot scale up the size of models by a thousand times again. No one is willing to spend billions of dollars on training runs yet.

From a hardware perspective, we should expect the pace of innovation to slow in the coming years.

Software advances are mixed. Using ML models is becoming easier by the day. With libraries like huggingface, a single line of code can run a state of the art model for your particular use case. There is a lot of room for software innovations to make it easier to use for non technical audiences, but right now very little research is bottlenecked by software.

Research advances are the X factor. Lots of people are working on these problems, and its possible there is a magic trick for intelligence at existing compute budgets. However, that is and always was true. However, the most important research advances of the last few years primarily enabled us to use more GPUs for a given problem. Now that we are starting to run up against the limits of data acquisition and monetary cost, less low hanging fruit is available.

(Side note: Even facebook has trouble training current state of the art models. Here are some chronicles of them trying to train a GPT-3 size model).

Conclusion.

I don't think we should expect performance gains in AI to accelerate over the next few years. As a researcher in the field, I expect the next few years will involve a lot of advances in the "long tail" of use cases and have less growth in the most studied areas. This is because we have achieved the easy pickings gains from hardware and software over the past decade.

This is my first time posting to lesswrong, and I decided to post a lightly edited first draft because if I start doing heavy edits I don't stop. Every time I see a very fast AGI prediction or someone claiming Moore's law will last a few more decades I start to write something, but this time I actually finished it before deciding to rewrite. As a result, it isn't an airtight argument, but more my general feelings as someone who has been at two of the top research institutions in the world.



Discuss

Reading the ethicists: A review of articles on AI in the journal Science and Engineering Ethics

18 мая, 2022 - 23:52
Published on May 18, 2022 8:52 PM GMT

Epistemic status: Stream of consciousness reactions to papers read in chronological order. Caveat lector.

I have a dirty family secret. My uncle is a professional ethicist.

In a not-too roundabout way, this is why I ended up looking at the October 2020 issue of the journal Science and Engineering Ethics, their special issue on the ethics of AI. I am now going to read that issue, plus every article this journal has published about AI since then [I wussed out and am just going to skim the latter for ones of special interest] and give you the deets.

October 2020Hildt et al., Editorial: Shaping Ethical Futures in Brain-Based and Artificial Intelligence Research

This is the introduction to the issue. They give each paper a sentence or two of summary and try to tie them all together. The authors helpfully give a list of topics they think are important:

• Data Concerns: Data management, data security, protection of personal data, surveillance, privacy, and informed consent.

• Algorithmic Bias and Discrimination: How to avoid bias and bias related problems? This points to questions of justice, equitable access to resources, and digital divide.

• Autonomy: When and how is AI autonomous, what are the characteristics of autonomous AI? How to develop rules for autonomous vehicles?

• Responsibility: Who is in control? Who is responsible or accountable for decisions made by AI?

• Questions relating to AI capabilities: Can AI ever be conscious or sentient? What would conscious or sentient AI imply?

• Values and morality: How to build in values and moral decision-making to AI? Are moral machines possible? Should robots be granted moral status or rights?

Based on this list, I anticipate that I'm about to run into four-sixths ethics papers about present-day topics that I will skim to point out particularly insightful or anti-insightful ones, one-sixth philosophers of mind that I will make fun of a little, and one-sixth papers on "How to build values into general AI" that I'm really curious as to the quality of.

Onward!

Nallur, Landscape of Machine Implemented Ethics

Primarily this paper is a review of a bunch of papers that have implemented or proposed ethics modules in AI systems (present-day things like expert systems to give medical advice, or lethal autonomous weapons [which he has surprisingly few qualms about]). These were mostly different varieties of rule-following or constraint-satisfaction, with a few handwritten utility functions thrown in. And then one of these is Stuart Armstrong (2015) for some reason - potentially that reason is that the author wanted to at least mention "value-loading," and nobody else was talking about it (I checked - there's a big table of properties of different proposals).

It also proposes evaluating different proposals by having a benchmark of trolley-problem-esque ethical dilemmas. The main reason this idea won't work is that making modern-day systems behave ethically involves a bunch of bespoke solutions only suitable to the domain of operation of that system, not allowing for cross-comparison in any useful way.

If were to salvage this idea, we might wish to have a big list of ethical questions the AI system should get the right answer to, and then when building a sufficiently important AI (still talking about present-day applications), the designers should go through this list and find all the questions that can be translated into their system's ontology and check that their decision-making procedure gets acceptable answers. E.g. "Is it better to kill one person or two people?" can become self-driving car scenarios where it's going to hit either one or two people, and it should get the right answer, but the self-driving car people don't have to benchmark their system on medical ethics questions.

Bauer, Expanding Nallur's Landscape of Machine Implemented Ethics

This paper makes the point that "ethical behavior" for present-day AI might also mean taking a stand about how we want society to be arranged on a larger scale (e.g. what's an "ethical high-frequency trading algorithm"?). Then it descends into self-advertisement about hedonistic utilitarianism and virtue ethics, which we should clearly not build AIs to use.

Farisco et al., Towards Establishing Criteria for the Ethical Analysis of Artificial Intelligence

This one makes a lot of the right noises at first. They spend a couple of pages talking about defining intelligence for some unclear reason, but at least they cite Legg and Hutter, you know? But then they quickly take a left turn off the deep end and start talking about how biological intelligence is morally different than AI because AI can't do abductive reasoning. There are some good points about emotional connection mixed in with the bad points about emotional reasoning being magic, I guess.

Butkus, The Human Side of Artificial Intelligence

This is a response to the previous article, especially the parts about emotional reasoning being magic. The key point it makes (a good one) is that humans aren't all that great at reasoning - we make lots of mistakes, including moral mistakes. "If we intentionally avoid some of the known pitfalls in our cognitive architecture, we cannot help but create moral agents that are dissimilar from us." How to do this in a trustworthy way is obviously hard, and they mumble something about contextuality of decision making.

Rainey and Erden, Correcting the Brain? The Convergence of Neuroscience, Neurotechnology, Psychiatry, and Artificial Intelligence

We might use AI to control brain stimulation to try to treat people with psychiatric problems. This would be like the AI controlling people, which is scary. Those darn neuroscientists are too reductive about the brain. Boo reductionism, yay holism. Humans look for and can relay bite-sized reasons for their actions, while AIs can't, which is why human reasoning is more trustworthy. </they say>

Jotterand and Bosco, Keeping the “Human in the Loop” in the Age of Artificial Intelligence

Props to this paper for having a good abstract. It's about risks from "dehumanizing" medicine. Their best point is that there's an inequality in the doctor-patient relationship, but that part of good medicine is establishing trust with the patient and genuinely working together with them to answer moral/medical questions. Of course they then say "AI will never ever be able to do this," but we can charitably interpret them as saying that AI can do it neither now nor soon, and that there are dangerous incentives to hastily replace doctors with AI in a way that damages patients' trust and agency.

Dubljević, Toward Implementing the ADC Model of Moral Judgment in Autonomous Vehicles

ADC stands for "Agent, Deed, Consequence." This would evaluate actions using a nearly-equal mixture of three parts. Normally the "Agent" part means that an action is more moral if the agent was a good person with good intentions, but in this paper the author also gives it the job of making actions more moral if you're helping good people or harming bad people. (Does this make any sense? Especially to program into a self-driving car? No, this seems like mixing up descriptive and normative.). "Deed" means checking if the action obeys the law or other pre-specified rules of the road. "Consequence" means not crashing or causing crashes, and getting to the destination.

The author gives a cherry-picked terrorism example where self-driving cars are supposed to notice that a terrorist is driving a truck into a crowd, judge that they're a bad person, and then will not get out of the truck's way because "Action" is negative (getting out of the way would help a bad person), "Deed" is negative (you'd have to leave your lane to avoid the truck, which is against the rules), and only "Consequence" is positive (you'd avoid crashing).

This isn't totally insane. We could imagine e.g. giving self-driving cars rules for identifying when another car is about to crash into people and how to try to intercept it. But I think this notion of needing a triumvirate where the three votes are Law, Not Crashing, and Harming Bad People is just the wrong design desiderata.

Totschnig, Fully Autonomous AI

Our first paper about ethics for par-human and superhuman AI! They care about something they call "full autonomy," where you sometimes rewrite your own goals, which humans sort of do (the example is someone who devotes themself to one cause but then later changes their mind). They then give a good summary of why Bostrom And The Gang think that an AI won't want to change its goals (which they call the Finality Argument that a self-improving AI's goals will remain fixed).

My first response was that humans only sort of have goals, and therefore only sort of rewrite them. This is anticipated in the next section, and the author basically says this is a good point, but they still think "full autonomy" is important and in some sense desirable. *shrug*

Their knock-down counterargument to the Finality Argument is that improving your model of the world requires you to translate your old goals into the new ontology, a task which is not specified by the goals themselves but by some extra standards (that may themselves be subject to change.) This is rewriting your goals, and so self-improving AIs have full autonomy as they define it.

All valid so far. But then they conclude "The good news is that the fear of a paper clip AI and similar monsters is unfounded. The bad news is that the hope of a human-equivalent or superhuman AI under our control, of a genie in a bottle, is unfounded as well." They think AIs are going to search for moral realism and end up having weird and surprising goals. But this makes an unsupported leap from the "full autonomy" they care about to "will radically rather than only subtly change itself."

One more thing I'm interested in here is to look through the references to see if there are any famous (well, at least well-cited) ethicists writing about things like this who I haven't heard of. A lot of the references were to the usual suspects (Bostrom, Yudkowsky, Russell, Yampolskiy, etc.). Papers for me to look into:

Lawless, W. F., Mittu, R., Sofge, D., & Russell, S. (Eds.). (2017). Autonomy and artificial intelligence: A threat or savior? (A volume of collected papers)

Redfeld, S. A., & Seto, M. L. (2017). Verification challenges for autonomous systems.

Tessier, C. (2017). Robots autonomy: Some technical issues.

Petersen, S. (2017). Superintelligence as superethical. In P. Lin, R. Jenkins, & K. Abney (Eds.), Robot ethics 2.0: From autonomous cars to artificial intelligence (pp. 322–337).

Podschwadek, F. (2017). Do androids dream of normative endorsement? On the fallibility of artificial moral agents. Artificial Intelligence and Law, 25(3), 325–339

I'll get back to these later.

Dennis, Computational Goals, Values and Decision‑Making

This is commentary on the previous paper. It mostly says "hold up, you can't actually implement an agent with a utility function, you need to use some kind of bounded rationality" without drawing strong conclusions.

Matthias, Dignity and Dissent in Humans and Non‑humans

This paper is another one that intensely cares about whether AIs (and animals) have autonomy and/or the ability to give themselves goals. But this time it explains some context: Kant cared a lot about human dignity and "derives it from the moral autonomy of the individual," and so people who've drunk the Kant kool-aid (probably could phrase that better, oh well) are asking if AIs have moral autonomy so they can know whether they're dignified.

However, this paper quickly decides that all that matters is autonomy similar in kind and quantity to that of a human, which they say isn't all that high a bar. However, they also say it isn't "real choice" if the AI is following steps for moral reasoning laid down by its programmers, which seems to mix up rule-goverenedness with human understanding of those rules (appeal to the mystique of mystery).

There's also something about individuality being important to them, which sort of makes sense but also sort of sounds like the author preferentially thinks about human-sized and human-shaped entities.

Zhu et al., Blame‑Laden Moral Rebukes and the Morally Competent Robot: A Confucian Ethical Perspective

Many of these authors have been from not-so-famous universities (I'm still used to the hard sciences where funding differential and race dynamics means that the same handful of institutions reliably dominate), but this one is surprising enough to mention: the authors are from the Colorado School of Mines, which I wasn't even aware had a school of humanities (they're a decent but very focused engineering school).

This paper is about an interesting question: should near-future language models (or AI systems that use a language model) output "blame-laden moral rebukes" of users who misbehave? If you start swearing at the AI receptionist, should it tell you off?

The authors first give some low-sample-size evidence that humans will be influenced by AI enforcement of moral norms (with different forms of enforcement, ranging from polite mentions to blunt demands, working better in different contexts). Then they spend a section explaining some facets of Confucianism that relate to when you should rebuke others, which point to some of the theory of mind an effectively-rebuking AI should have. They also advocate for some Confucian norms about filtering communication through social role.

Gunkel, Shifting Perspectives

This is commentary on the previous article. It's not very good, but does have some nice references if you want to look into the subfield called Human-Machine Communication.

Aicardi et al., Ethical and Social Aspects of Neurorobotics

This article comes out of the Human Brain Project, an ambitious (and non-ambition-meeting) effort to scan the human brain. After getting what felt like boilerplate social concerns out of the way, they share some of the things that they had to think about the ethics of at HBP. The key sections were on the dual use potential of robotics research, the issues with academic-industry partnerships, and managing and securing your data.

Taraban, Limits of Neural Computation in Humans and Machines

This was supposed to be commentary on the previous article, except the author didn't want to talk about research ethics, they wanted to talk about whether it makes sense to build robots using AI derived from scans of human brains.

First, they say it's silly in the first place. Humans are flawed thinkers, and anyhow we don't know how to take a brain scan and turn it into an AI with customized semantic content, and anyhow the probabilistic inference people are probably going to build better robots first.

Second, if the neurorobotics people succeed we'd probably grant the robots rights, and that would be silly so let's not do it.

Soltanzadeh et al., Customizable Ethics Settings for Building Resilience and Narrowing the Responsibility Gap

This is an article about autonomous vehicles, arguing that they should come with user-modifiable "ethics settings." These could be things like setting the tradeoff between speed and greenhouse gas emissions, adjusting car follow distance and pedestrian avoidance distance within some range, etc.

Basically they call user autonomy a "core social value," and also worry that if the human in the vehicle isn't responsible for what the car does, that's inherently bad. The weirdness of these arguments actually makes me re-examine my intuitive agreement with the idea of ethics settings.

Ryan, In AI We Trust: Ethics, Artificial Intelligence, and Reliability

The EU's HLEG AI Ethics guidelines state that humans should be able to trust AI. The author splits hairs, gives some definitions of trust that require emotional states or motivations on the part of the person being trusted, and then concludes that we don't actually want to be able to trust AI, we just want to be able to rely on it. *shrug*

Smids, Danaher’s Ethical Behaviourism: An Adequate Guide to Assessing the Moral Status of a Robot?

This is a response to an apparently-popular paper, Danaher 2019, that argues for "ethical behaviorism" for robots - if they act like an animal, then we should ascribe animal-like moral status to them. The author (Smids) disagrees.

This paper could really use more Bayesian reasoning. They try to muddle through based on what is qualitatively "justified" in abductive reasoning, but it gets real murky.

The point the author makes is: ethical behaviorism is sneaking in a theory about what makes things have moral status, and if we just use that theory we don't need to keep the behaviorism part.

For example, breathing is a behavior that many animals of high moral status do, but we don't ask a robot to breathe before it can have moral status. But how do we know to discard breathing, if we're just being behaviorists? We're discriminating between different kinds of behavior by checking how much information they give us about the morally significant properties (e.g. feeling pain, having dreams) that we actually care about.

And then once you allow that we're using some theory about mental faculties and inferring them from behavior, it makes sense to allow us to infer them from other things too - surely the design of a robot gives us more than literally zero additional information about its mental faculties.

I think this paper was reasonable and well argued, but I think it also illuminates some issues in ethicists' approach to AI. First, Danaher's paper feels like it might have been popular because of the "style points for cleverly defending something silly" dynamic that's occasionally a problem, plus some wishful thinking for simple solutions. Second, both the original paper and the response focus on animal-like or human-like AI controlling individual robot bodies, a very anthropomorphic (zoomorphic?) picture that seems to show up repeatedly whenever people first try to think about the "moral worth of AI."

Cawthorne and van Wysnberghe, An Ethical Framework for the Design, Development, Implementation, and Assessment of Drones Used in Public Healthcare

This paper foresees an expansion of drone usage by healthcare organizations and tries to go through ethical principles (beneficence, non-maleficence, autonomy, justice, explicability) to try to draw recommendations. Among many recommendations of varying quality, some of the interesting ones were:

  • Deliberately limiting the drone's capabilities to only what's needed for its specific job (avoiding excess capability)
  • Taking steps to make sure that it's really hard to use them for surveillance
  • Flying them along designated corridors rather than the shortest route if that aids predictability and helps avoid crashes
  • Making sure they're quiet
  • Technological transparency
Later Papers I Found InterestingHubbard and Greenblum, Surrogates and Artificial Intelligence: Why AI Trumps Family (December 2020)

When someone is incapacitated, right now we let their family make medical decisions for them. Instead, we should train a model of medical decision-making on the choices of non-incapacitated patients, conditional on various demographic observables (age, sex, etc.) and use the predicted decision. If this model is more accurate at predicting the person's preferences than the family, then it's more ethical to use the model than to listen to the family.

A+ provocative paper.

Ryan et al., Research and Practice of AI Ethics: A Case Study Approach (March 2021)

Half the reason I include this is that they do devote a paragraph or two to "long-term issues" like superintelligent AI. But then they go back to looking for ethical issues in present-day case studies, and find only present-day type issues. Still, this was actually a pretty decent way of getting at peoples' concerns about present-day uses of AI and big data, which is the other half a reason to mention this paper.

Mamak, Rights for Robots: Artificial Intelligence, Animal and Environmental Law (2020) by Joshua Gellers (April 2021)

This is just a reminder that "Will robots have rights?" is absolute catnip to many ethicists. I will no longer be including more "But what about robot rights" papers, but you can be assured that they exist.

Lara, Why a Virtual Assistant for Moral Enhancement When We
Could have a Socrates? (June 2021)

I think I've actually skimmed this paper before. Or maybe the author's been shopping this idea around for a while.

As the title says, the author proposes a method (the method, they say) for augmenting humans morally, which is to help them reflect with non-invasive tools or partners that focus on improving their reasoning skills, not on leading them to one particular conclusion. Such a tool, let's call it "SocrAItes," might be possible quite soon, based on technical achievements like GPT3 or IBM's Project Debater.

I think it's an interesting idea that someone genuinely should try out, but I'm not sold that all other ideas are bad. Also, the author doesn't really think through how one would design or train such a system, so if you wanted to take a crack at it you'll need to start from square one.

de Sio, Mark Coeckelbergh, AI Ethics, Mit Press, 2021 (August 2021)

This book review wasn't that great, but it did make me go look up the original to see what Coeckelbergh had to say about superintelligent AI. Sadly, his chapter on superintelligence (the first chapter of the book) is spent rehashing Frankenstein and the myth of the golem, rather than talking about the ethics of superintelligent AI. Then the next chapter is spent resurrecting Hubert Dreyfus to argue against the possibility of general AI (plus a mildly interesting discussion of humanism, transhumanism, and anti-humanistic posthumanism.) But we get down to brass tacks in chapter 3, which is about whether robots should have rights.

Roberts et al., Achieving a ‘Good AI Society’: Comparing the Aims
and Progress of the EU and the US (November 2021)

An interesting summary paper if you want to read about international AI governance aimed at present-day issues. However, actually getting into the details of empirical questions like how the EU's AI regulations are actually being enforced seems like it's difficult and requires more data than is used in this paper - this paper mostly just covers the aims of the EU and US.

The gist of their picture is that the EU is trying to address a broad set of risks from current AI systems, and the US is trying to address a much more narrow set and is pressuring other countries to do the same because addressing more risks would cut into US company profits.

Schmid et al., Dual‑Use and Trustworthy? A Mixed Methods Analysis of AI
Diffusion Between Civilian and Defense R&D (January 2022)

This is an interesting attempt to characterize the dual-use of AI technology by looking at patent citations. Among 109 AI patent citations between different companies in Germany from 2008 to 2018, 93 stayed between civilian companies, 12 were a civilian company being cited by a defense company, 3 were a defense company cited by a civilian company, and 1 was between defense companies.

Which is interesting and all, but they don't actually do a good enough job of checking what this means for dual use (they say it's not happening). Like, how does this compare to citation patterns for technologies that more clearly are / are not dual use? Overall grade: cool idea, but I still have as many questions as I did at the start.

Conclusions

I did this so you don't have to. Out of the 45+ papers I looked through, I would say to read Nallur (2020) to get a survey of present-day machine ethics work, read Roberts et al. (2021) if you're interested in AI governance, and forward Hubbard and Greenblum (2020) to your doctor friends because it's a great troll.

There were fewer papers than I expected focused on the ethics of superhuman AI, though a decent number mentioned the issue (citing Bostrom, not really each other). However, I have found some good papers on the ethics of par-human or superhuman AI outside the journal Science and Engineering Ethics, which I'll cover in the sequel to this post. I'm not sure why this is - it could be that the fraction of ethics papers on superintelligence is constant and I merely found them effectively when I searched for them, or there was a bump in interest after the publication of Superintelligence: Paths, Dangers, Strategies that has now subsided, or this is due to a feature of the journal like its culture or an opinion of the editors.

What do I want to see in relation to ethicists? I don't think you can take someone who currently thinks about the average of the papers above, and immediately get them to have interesting thoughts about superhuman AI. But people can learn new things, and people interested in ethics are more or less the sort of people who'll get interested in the ethics of superhuman AI. So I would recommend more high-quality papers making the basic arguments in new ways or showing off incremental progress in a way accessible to the ethics or philosophy communities, but not recommend rapid incentives for new papers from people who haven't "put in the time."

One place that the current literature does seem relatively expert (and relatively influential) is in present-day governance of AI. I think that people working on AI governance with an eye towards risks from advanced AI should absolutely be trying to work in partnership with the broader AI regulatory community.



Discuss

Maxent and Abstractions: Current Best Arguments

18 мая, 2022 - 22:54
Published on May 18, 2022 7:54 PM GMT

This post is not-very-distilled and doesn’t contain much background; it’s intended for people who already have the context of at least these four posts. I’m putting it up mainly as a reference for people who might want to work directly on the math of natural abstractions, and as a technical reference post.

There’s various hints that, in most real-world cases, the distribution of low-level state given high-level natural abstractions should take the form of a maximum entropy distribution, in which:

  • The “features” are sums over local terms, and
  • The high-level variables are (isomorphic to) the Lagrange multipliers

More formally: we have a low-level causal model (aka Bayes net) P[XL]=∏iP[XLi|XLpa(i)].mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} . Given the high-level variables XH, the distribution of low-level variable values should look like

P[XL|XH]=1ZP[XL]eλT(XH)∑ifi(XLi,XLpa(i))

… i.e. the maximum-entropy distribution subject to constraints of the form E[∑ifi(XLi,XLpa(i))|XH]=μ(XH). (Note: λ, fi, and μ are all vector-valued.)

This is the sort of form we see in statistical mechanics. It’s also the form which the generalized Koopman-Pitman-Darmois (gKPD) theorem seems to hint at.

I don’t yet have a fully-satisfying general argument that this is the main form which abstractions should take, but I have two partial arguments. This post will go over both of them.

Maxent Telephone Theorem ArgumentTwo different nested layers of Markov blankets on the same underlying causal DAG

Quick recap of the Telephone Theorem: information about some variable X passes through a nested sequence of Markov blankets M1,M2,…. Information about X can only be lost as it propagates. In the limit, all information is either perfectly conserved or completely lost. Mathematically, in the limit P[X|Mn]=P[X|Fn(Mn)] for some F such that Fn(Mn)=Fn+1(Mn+1) with probability approaching 1 as n→∞; F is the perfectly-conserved-in-the-limit information carrier.

In this setup, we can also argue that the limiting distribution limn→∞P[X|Mn] should have a maxent form. (Note: this is a hand-wavy argument, not a proper proof.)

Think about how the distribution (x↦P[X=x|Mn]) transforms as we increment n by 1. We have

P[X|Mn+1]=∑MnP[X|Mn]P[Mn|Mn+1]

First key property of this transformation: it’s a convex combination for each Mn+1 value, i.e. it’s mixing. Mixing, in general, cannot decrease the entropy of a distribution, only increase it or leave it the same. So, the entropy of P[X|Mn] will not decrease with n.

When will the entropy stay the same? Well, our transformation may perfectly conserve some quantities. Since the transformation is linear, those quantities should have the form ∑Xf(X)P[X|Mn] for some f, i.e. they’re expected values. They’re conserved when E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1.

Intuitively, we’d expect the entropy of everything except the conserved quantities to strictly increase. So, we’d expect the distribution P[X|Mn] to approach maximum entropy subject to constraints of the form E[f(X)|Mn]=μ(Mn), where E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1 (at least in the limit of large n). Thus, we have the maxent form

P[X|Mn]=1ZP[X]eλT(Mn)f(X)

(Note on the P[X] in there: I’m actually maximizing relative entropy, relative to the prior on X, which is almost always what one should actually do when maximizing entropy. That results in a P[X] term. We should find that E[lnP[X]|Mn] is a conserved quantity anyway, so it shouldn’t actually matter whether we include the P[X] multiplier or not; we’ll get the same answer either way.)

Shortcomings of This Argument

Obviously it’s a bit handwavy. Other than that, the main issue is that the Telephone Theorem doesn’t really leverage the spatial distribution of information; information only propagates along a single dimension. As a result, there’s not really a way to talk about the conserved f’s being a sum over local terms, i.e. f(X)=∑ifi(Xi,Xpa(i)).

Despite the handwaviness, it’s an easy result to verify computationally for small systems, and I have checked that it works.

Resampling + gKPD Argument

Another approach is to start from the redundancy + resampling formulation of abstractions. In this approach, we run an MCMC process on our causal model. Any information which is highly redundant in the system - i.e. the natural abstractions - is near-perfectly conserved under resampling a single variable at a time; other information is all wiped out. Call the initial (low-level) state of the MCMC process X0, and the final state X. Then we have

P[X|X0]=P[X|F(X0)]=P[X|F(X)]P[F(X)|F(X0)]=1ZP[X]I[F(X)=F(X0)]

… where F is conserved by the resampling process with probability 1.

It turns out that P[X|X0] factors over the same DAG as the underlying causal model:

P[X|X0]=∏iP[Xi|Xpa(i),X0]

If the conserved quantities F(X) are much lower-dimensional than X itself, then we can apply the gKPD theorem: we have a factorization of P[X|X0], we have a low-dimensional summary statistic F(X) which summarizes all the info in X relevant to X0, so the gKPD theorem says that the distribution must have the form

P[X|X0]=1ZeλT(X0)∑i∉Efi(Xi,Xpa(i))∏i∉EP[Xi|Xpa(i),X0=(X0)∗]∏i∈EP[Xi|Xpa(i),X0=X0]

… where E is a relatively-small set of “exceptional” indices, and (X0)∗ is some fixed reference value of X0. This is slightly different from our intended form - there’s the exception terms, and we have ∏i∉EP[Xi|Xpa(i),X0=(X0)∗] rather than just ∏i∉EP[Xi|Xpa(i)]. The latter problem is easily fixed by absorbing ∏i∉EP[Xi|Xpa(i),X0=(X0)∗]P[Xi|Xpa(i)] into f (at the cost of possibly increasing the summary dimension by 1), so that’s not really an issue, but the exception terms are annoying. Absorbing and assuming (for convenience) no exception terms, we get the desired form:

P[X|X0]=1ZeλT(X0)∑ifi(Xi,Xpa(i))P[X]

Note that this is maxentropic subject to constraints of the form E[∑ifi(Xi,Xpa(i))|X0]=μ(X0). Since the summary statistic F(X)=∑ifi(Xi,Xpa(i)) is conserved by the resampling process, we must have μ(X0)=∑ifi(X0i,X0pa(i)), so the conservation equation is

E[∑ifi(Xi,Xpa(i))|X0]=∑ifi(X0i,X0pa(i))

Shortcomings of This Argument

Obviously there’s the exception terms. Other than that, the main issue with this argument is an issue with the resampling approach more generally: once we allow approximation, it’s not clear that the natural abstractions from the resampling formulation are the same natural abstractions which make the Telephone Theorem work. Both are independently useful: information dropping to zero at a distance is an easy property to leverage for planning/inference, and knowing the quantities conserved by MCMC makes MCMC-based planning and inference much more scalable. And in the limit of perfect conservation and infinite “distance”, the two match. But it’s not clear whether they match under realistic approximations, and I don’t yet have efficient methods to compute the natural abstractions both ways in large systems in order to check.

That said, resampling + gKPD does give us basically the result we want, at least for redundancy/resampling-based natural abstractions.



Discuss

Framing Practicum: Dynamic Programming

18 мая, 2022 - 21:16
Published on May 18, 2022 6:16 PM GMT

This is a framing practicum post. We'll talk about what dynamic programming (DP) is, how to recognize DP in the wild, and what questions to ask when you find it. Then, we'll have a challenge to apply the idea.

Today's challenge: come up with 3 examples of DP which do not resemble any you've seen before. They don't need to be good, they don't need to be useful, they just need to be novel (to you).

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What is DP?

Suppose I am about to drive from Miami to Boston and I need to get to Boston as fast as possible. As a first step, I check the highway map and create a list of possible routes for this trip (let’s assume “good” old times with no Google maps). For instance, looking at the imaginary routes in the figure below, the route at the top says I should take the “Miami → A → C → E → G → Boston” route and the total trip distance would be 200+170+220+400+430=1420 miles. Which specific route should I take to minimize the total distance, thus total travel time? I can, of course, calculate total travel distance for each possible route and pick the one with least-distance. But it could easily get very time consuming if there exist hundreds of thousands of possible routes to evaluate.

One alternative approach to identify a route with minimum distance is to use a backward method. Suppose I drive backward through the route map from right to left. First, I will start at the destination, Boston. If I am in city G or H, I have no further decision to make since there is only one route that leads me to Boston from either city. The number in green summarizes total distance with one last trip, or one stage, to go.

My first decision (from right to left) occurs with two trips, or stages to go. If, for example, I am in city F, I can either drive 150 miles to city G and another 430 miles to Boston - total 580 miles, or drive 200 miles to city H and another 400 miles to Boston - total 600 miles. Therefore, the shortest possible distance, or optimal route (in green), from city F is 580 miles (F → G → Boston). The alternative route from F (F → H → Boston) is suboptimal with 600 miles to go (in red). 

Let me back up one more city, or stage, and compute the least-distance from city C and D to Boston. Figure below summarizes these calculations.

Once I have computed the optimal route from city C and D onward to Boston, I can again move back one city and determine the optimal route from city A and B onward.

I continue this process and will end up with an optimal route with least-distance of 1080 miles to the problem (highlighted in bold arrows):

This is DP: An agent faces a multistage optimization problem (travelling from Miami to Boston by travelling through multiple cities). At each stage (e.g., I have one more trip to go to Boston), the agent might be in a different state (e.g., I am currently in city A or city B). According to the current state (e.g., I am in city A), the agent takes a specific action (e.g., I will drive to city C) and as a result of that action, the system transitions to a different state in the next stage (e.g., now I am in city C). We solve the multistage optimization problem by working backwards, at each step computing the best reward we can get from that stage onward by considering each of the possible “next/child” stages.

What To Look For?

DP should come to mind whenever an agent faces a problem with multi-stages nature and the agent takes a series of actions. Another defining feature of DP is that the original multi-stage complex problem can be dismantled into a sequence of simpler and smaller problems. The action the agent takes in a particular stage depends on the current state and the reward the agent would receive by taking that specific action. In addition, the action the agent takes impacts the state of the system, causing the system to transition to a new state.

Useful Questions to Ask?

In the shortest driving time example, the ultimate goal is to minimize total driving time such that I can arrive at Boston as fast as possible. At any given time in my trip, I might be in a different state - the city I am in at that time. For instance, on the second day of the trip, I might be in city C or city D. Given my state,  I have a simpler problem to solve: What is the shortest travel time from city C or city D to Boston?

The system may start in different states. The agent takes a series of actions to optimize an objective across multiple stages. Each stage also has multiple states. The specific action an agent can take is a function of the state the agent currently in.

In general, whenever we see problems where DP is applicable, we should ask:

  • What is the objective?
  • What are the stages and states of the system? 
  • What are the actions the agent can take at any given state?
  • How does a specific action change the state of the system?
  • What is the value function? How is an agent rewarded for taking a particular action at the current stage?
The Challenge

Come up with 3 examples of incentives which do not resemble any you’ve seen before. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Any answer must include at least 3 to count, and they must be novel to you. That’s the challenge. We’re here to challenge ourselves, not just review examples we already know.

However, they don’t have to be very good answers or even correct answers. Posting wrong things on the internet is scary, but a very fast way to learn, and I will enforce a high bar for kindness in response-comments. I will personally default to upvoting every complete answer, even if parts of it are wrong, and I encourage others to do the same.

Post your answers inside of spoiler tags. (How do I do that?)

Celebrate others’ answers. This is really important, especially for tougher questions. Sharing exercises in public is a scary experience. I don’t want people to leave this having back-chained the experience “If I go outside my comfort zone, people will look down on me”. So be generous with those upvotes. I certainly will be.

If you comment on someone else’s answers, focus on making exciting, novel ideas work — instead of tearing apart worse ideas. Yes, And is encouraged.

I will remove comments which I deem insufficiently kind, even if I believe they are valuable comments. I want people to feel encouraged to try and fail here, and that means enforcing nicer norms than usual. 

If you get stuck, look for:

  • Problems with multistage nature.
  • Problems that can be dismantled into a sequence of simpler problems.

Bonus Exercise: for each of your three examples from the challenge, explain:

  • What are the simpler and smaller problems in the DP example?
  • What are the states and how taking a specific action alters the state?
  • what is the reward for taking a specific action on a given state?

This bonus exercise is great blog-post fodder!

Motivation

Using a framing tool is sort of like using a trigger-action pattern: the hard part is to notice a pattern, a place where a particular tool can apply (the “trigger”). Once we notice the pattern, it suggests certain questions or approximations (the “action”). This challenge is meant to train the trigger-step: we look for novel examples to ingrain the abstract trigger pattern (separate from examples/contexts we already know).

The Bonus Exercise is meant to train the action-step: apply whatever questions/approximations the frame suggests, in order to build the reflex of applying them when we notice incentives.

Hopefully, this will make it easier to notice when an incentive frame can be applied to a new problem you don’t understand in the wild, and to actually use it.



Discuss

How to get into AI safety research

18 мая, 2022 - 21:05
Published on May 18, 2022 6:05 PM GMT

Recently, I had a conversation with someone from a math background, asking how they could get into AI safety research. Based on my own path from mathematics to AI alignment, I recommended the following sources. It may prove useful to others contemplating a similar change in career:

  • Superintelligence by Nick Bostrom. It condenses all the main arguments for the power and the risk of AI, and gives a framework in which to think of the challenges and possibilities.
  • Sutton and Barto's Book: Reinforcement Learning: An Introduction. This gives the very basics of what ML researchers actually do all day, and is important for understanding more advanced concepts. It gives (most of) the vocabulary to understand what ML and AI papers are talking about.
  • Gödel without too many tears. This is how I managed to really grok logic and the completeness/incompleteness theorems. Important for understanding many of MIRI's and LessWrong's approaches to AI and decision theory.
  • Safely Interruptible agents. It feels bad to recommend one of my own papers, but I think this is an excellent example of bouncing between ML concepts and alignment concepts to make some traditional systems interruptible (so that we can shut them down without them resisting the shutdown).
  • Alignment for Advanced Machine Learning Systems. Helps give an overall perspective on different alignment methods, and some understanding of MIRI's view on the subject (for a deeper understanding, I recommend diving into MIRI's or Eliezer's publications/writings).

You mileage may vary, but these are the sources that I would recommend. And I encourage you to post any sources you'd recommend, in the comments.



Discuss

A bridge to Dath Ilan? Improved governance on the critical path to AI alignment.

18 мая, 2022 - 18:51
Published on May 18, 2022 3:51 PM GMT

Summary: This post showcases my finalist entry in the Future of Life Institute's AI worldbuilding contest.  It imagines:

  1. How we might make big improvements to decisionmaking via mechanisms like futarchy and liquid democracy, enhanced by Elicit-like research/analysis tools.
  2. How changes could spread to many countries via competition to achieve faster growth than rivals, and via snowball effects of reform.
  3. How the resulting, more "adequate" civilization could recognize the threat posed by alignment and coordinate to solve the problem.

(Cross-posted to the EA Forum)

Part of a mural illustrating our scenario, created by Diana Gurvich!Motivation for our scenario:

Human civilization's current ability to coordinate on goals, make wise decisions quickly, and capably execute big projects, seems inadequate to handle the challenge of safely developing aligned AI.  Evidence for this statement can be found practically all around you, but the global reaction to covid-19 is especially clarifying.  Quoting Gwern:

The coronavirus was x-risk on easy mode: a risk (global influenza pandemic) warned of for many decades in advance, in highly specific detail, by respected & high-status people like Bill Gates, which was easy to understand with well-known historical precedents, fitting into standard human conceptions of risk, which could be planned & prepared for effectively at small expense, and whose absolute progress human by human could be recorded in real-time happening rather slowly over almost half a year while highly effective yet cheap countermeasures like travel bans & contact-tracing & hand-made masks could—and in some places did!—halt it. Yet, most of the world failed badly this test; and many entities like the CDC or FDA in the USA perversely exacerbated it, interpreted it through an identity politics lenses in willful denial of reality, obstructed responses to preserve their fief or eek out trivial economic benefits, prioritized maintaining the status quo & respectability, lied to the public “don’t worry, it can’t happen! go back to sleep” when there was still time to do something, and so on. If the worst-case AI x-risk happened, it would be hard for every reason that corona was easy. When we speak of “fast takeoffs”, I increasingly think we should clarify that apparently, a “fast takeoff” in terms of human coordination means any takeoff faster than ‘several decades’ will get inside our decision loops. Don’t count on our institutions to save anyone: they can’t even save themselves.

Around here on LessWrong, proposed AI x-risk-mitigation strategies generally attempt to route around this problem by aiming to first invent an aligned superintelligent AI, then use the superintelligent AI to execute a "pivotal action" that prevents rival unaligned AIs from emerging and generally brings humanity to a place of existential security.

This is a decent Plan A -- it requires solving alignment, but we have to solve that eventually in almost every successful scenario (including mine).  It doesn't require much else, making it a nice and simple plan.  One problem might be that executing a massive "pivotal action" might work less well if AI capabilities develop more smoothly and capabilities are distributed evenly among many actors, a la "slow takeoff" scenarios.

But some have argued have argued that we might be neglecting "Plan B" strategies built around global coordination.  The post "What An Actually Pessimistic Containment Strategy Looks Like" considers Israel's successful campaign to stop Iran from developing nuclear weapons, and argues that activist efforts to slow down AGI research at top tech companies might be similarly fruitful.  Usually (including in my worldbuilding scenario), it's imagined that the purpose of such coordination is to buy a little more time for technical alignment safety work to happen.  But for a more extreme vision of permanently suppressing AI technology, we can turn to the fictional world of Dath Ilan, or to Nick Bostrom's "easy nukes" thought experiment exploring how humanity could survive if nuclear weapons were absurdly easy to make.

The idea that we should push for improved governance in order to influence AI has its problems.  It takes a long time, making it might be very helpful in 2070 but not by 2030.  (In this respect it is similar to other longer-term interventions like gene-editing to create more scientific geniuses or general EA community-building investments.)  And of course you still have to solve the technical challenge of AI alignment in the end.  But improving governance also has a lot to recommend it, and it's something that can ideally be done in parallel with technical alignment research -- complementing rather than substituting, worked on by different people who have different strengths and interests.

Finally, another goal of the story was expressing the general value of experimentation and governance competition.  I think that existing work in the cause area of "improving institutional decisonmaking" too heavily focused on capturing the commanding heights of existing prestigious institutions and then implementing appropriate reforms "from the inside".  This is good, but it too should be complemented by the presence of more radical small-scale experimentation on the "outside" -- things like charter cities and experimental intentional communities -- which could test out wildly different concepts of ideal governance.

Below, I've selected some of the most relevant passages from my contest submission.  To get more of the sci-fi utopian flavor of what daily life would be like in the world I'm imagining (including two wonderful short stories written by my friend Holly, a year-by-year timeline, and more), visit the full page here.  Also, the Future of Life Institute would love it if you submitted feedback on my world and the other finalists -- how realistic do you find this scenario, how much would you enjoy living in the world I describe, and so forth.

Excerpts from my team's contest submissionIllustrating governance innovation, the Flash Crash War, the Delhi Accords & subsequent golden age.Artificial General Intelligence (AGI) has existed for at least five years but the world is not dystopian and humans are still alive! Given the risks of very high-powered AI systems, how has your world ensured that AGI has at least so far remained safe and controlled?

Ultimately, humanity was able to navigate the dangers of AGI development because the early use of AI to automate government services accidentally kicked off an “arms race” for improved governance technology and institution design. These reforms improved governments’ decision-making abilities, enabling them to recognize the threat posed by misalignment and coordinate to actually solve the problem, implementing the “Delhi Accords” between superpowers and making the Alignment Project civilization’s top priority.

In a sense, all this snowballed from a 2024 Chinese campaign to encourage local governments to automate administrative processes with AI. Most provinces adopted mild reforms akin to Estonia’s e-governance, but some experimented with using AI economic models to dynamically set certain tax rates, or using Elicit-like AI research-assistant tools to conduct cost-benefit analyses of policies, or combining AI with prediction markets. This goes better than expected, kickstarting a virtuous cycle:

  • Even weak AI has a natural synergy with many government functions, since it makes predicting / planning / administering things cheap to do accurately at scale.
  • Successful reforms are quickly imitated by competing regions (whether a neighboring city or a rival superpower) seeking similar economic growth benefits.
  • After adopting one powerful improvement to fundamental decisionmaking processes, it’s easier to adopt others (ie, maybe the new prediction market recommends switching the electoral college to a national-popular-vote with approval voting).

One thing leads to another, and soon most of the world is using a dazzling array of AI-assisted, prediction-market-informed, experimental institutions to govern a rapidly-transforming world.

 

The dynamics of an AI-filled world may depend a lot on how AI capability is distributed. In your world, is there one AI system that is substantially more powerful than all others, or a few such systems, or are there many top-tier AI systems of comparable capability?

Through the 2020s, AI capabilities diffused from experimental products at top research labs to customizable commercial applications much as they do today. Thus, new AI capabilities steadily advanced through different sectors of economy.

The 2030s brought increasing concern about the power of AI systems, including their military applications. Against a backdrop of rapidly improving governance and a transforming international situation, governments started rushing to nationalize most top research organizations, and some started to restrict supercomputer access. Unfortunately, this rush to monopolize AI technology still paid too little attention to the problem of alignment; new systems were deployed all the time without considering the big picture.

After 2038’s Flash Crash War, the world woke up to the looming dangers of AGI, leading to much more comprehensive consolidation. With the Delhi Accords, all top AI projects were merged into an internationally-coordinated Apollo-Program-style research effort on alignment and superintelligence. Proliferation of advanced AI research/experimentation outside this official channel is suppressed, semiconductor supply chains are controlled, etc. Fortunately, the world transitioned to this centralized a few years before truly superhuman AGI designs were discovered.

As of 2045, near-human and “narrowly superhuman” capabilities are made broadly available through API for companies and individuals to use; hardware and source code is kept secure. Some slightly-superhuman AGIs, with strict capacity limits, are being cautiously rolled out in crucial areas like medical research and further AI safety research. The most cutting-edge AI designs exist within highly secure moonshot labs for researching alignment.

 

How has your world avoided major arms races and wars?

Until 2038, geopolitics was heavily influenced by arms races, including the positive "governance arms race" described earlier. Unfortunately, militaries also rushed to deeply integrate AI. The USA & China came to the brink of conflict during the “Flash Crash War”, when several AI systems on both sides of the South China Sea responded to ambiguous rival military maneuvers by recommending that their own forces be deployed in a more aggressive posture. These signaling loops between rival AI systems lead to an unplanned, rapidly escalating cycle of counter-posturing, with forces being rapidly re-deployed, in threatening and sometimes bizarre ways. For about a day, both countries erroneously believed they were being invaded by the other, leading to intense panic and confusion until the diplomatic incident was defused by high-level talks.

Technically, the Flash Crash War was not caused by misalignment per se (rather, like the 2010 financial Flash Crash, by the rapid interaction of multiple complex automated systems). Nevertheless, it was a fire-alarm-like event which elevated "fixing the dangers of AI systems" to a pressing #1 concern among both world leaders and ordinary people.

Rather than the lukewarm, confused response to crises like Covid-19, the world's response was strong and well-directed thanks to the good-governance arms race. Prediction markets and AI-assisted policy analysts quickly zeroed in on the necessity of solving alignment. Adopted in 2040, the Delhi Accords began an era of intensive international cooperation to make AI safe. This put a stop to harmful military & AI-technology arms races.
 

In the US, EU, and China, how and where is national decision-making power held, and how has the advent of advanced AI changed that?

The wild success of China's local-governance experiments led to freer reign for provinces. Naturally, each province is very unique, but each now uses AI to automate basic government services, and advanced planning/evaluation assistants to architect new infrastructure and evaluate policy options.

The federal government's remaining responsibilities include foreign relations and coordinating national projects. The National People's Congress now mostly performs AI-assisted analysis of policies, while the Central Committee (now mostly provincial governors) has regained its role as the highest governing body.

In the United States, people still vote for representatives, but Congress debates and tweaks a basket of metrics rather than passing laws or budgets directly. This weighted index (life expectancy, social trust, GDP, etc) is used to create prediction markets where traders study whether a proposed law would help or hurt the index. Subject to a handful of basic limits (laws must be easy to understand, respect rights, etc), laws with positive forecasts are automatically passed.
This system has extensively refactored US government, creating both wealth and the wisdom needed to tackle alignment.

The EU has taken a cautious approach, but led in other areas:

  • Europe has created an advanced hybrid economy of "human-centered capitalism", putting an automated thumb on the scale of nearly every transaction to favor richer social connections and greater daily fulfillment.
  • Europe has also created the most accessible, modular ecosystem of AI/governance tech for adoption by other countries. Brazil, Indonesia, and others have benefited from incorporating some of the EU's open-source institutions.

 

What changes to the way countries govern the development, deployment and/or use of emerging technologies (including AI) played an important role in the development of your world?

After the world woke up to the dangers of powerful misaligned AI in 2038, nations realized that humanity is bound together by the pressing goal of averting extinction. Even if things go well, the far-future will be so strange and wonderful that the political concept of geopolitical “winners” and “losers” is impossible to apply.

This situation, like a Rawlsian veil of ignorance, motivated the superpowers to cooperate with 2040 Delhi Accords. Key provisions:

  • Nationalizing and merging top labs to create the Alignment Project.
  • Multi-pronged control of the “AI supply chain” (inspired by uranium & ICBM controls) to enforce nonproliferation of powerful AI — nationalizing semiconductor factories and supercomputer clusters, banning dangerous research, etc.
  • Securing potential attack vectors like nuclear command systems and viral synthesis technology.
  • API access and approval systems so people could still develop new applications & benefit from prosaic AI.
  • Respect for rights, plus caps on inequality and the pace of economic growth, to ensure equity and avoid geopolitical competition.

Although the Accords are an inspiring achievement, they are also provisional by design: they exist to help humanity solve the challenge of developing safe superintelligent machines. The Alignment Project takes a multilayered approach -- multiple research teams pursue different strategies and red-team each other, layering many alignment strategies (myopic oracle wrappers, adversarial AI pairs, human-values-trained reward functions, etc). With luck, these enable a “limited” superintelligence not far above human abilities, as a tool for further research to help humanity safely take the next step.

 

What is a new social institution that has played an important role in the development of your world?

New institutions have been as impactful over recent decades as near-human-level AI technology. Together, these trends have had a multiplicative effect — AI-assisted research makes evaluating potential reforms easier, and reforms enable society to more flexibly roll out new technologies and gracefully accommodate changes. Futarchy has been transformative for national governments; on the local scale, "affinity cities" and quadratic funding have been notable trends.

In the 2030s, the increasing fidelity of VR allows productive remote working even across international and language boundaries. Freed from needing to live where they work, young people choose places that cater to unique interests. Small towns seeking growth and investment advertise themselves as open to newcomers; communities (religious beliefs, hobbies like surfing, subcultures like heavy-metal fans, etc) select the most suitable town and use assurance contracts to subsidize a critical mass of early-adopters to move and create the new hub. This has turned previously indistinct towns to a flourishing cultural network.

Meanwhile, Quadratic Funding (like a hybrid of local budget and donation-matching system, usually funded by land value taxes) helps support community institutions like libraries, parks, and small businesses by rewarding small-dollar donations made by citizens.

The most radical expression of institutional experimentation can be found in the constellation of "charter cities" sprinkled across the world, predominantly in Latin America, Africa, and Southeast Asia. While affinity cities experiment with culture and lifestyle, cities like Prospera Honduras have attained partial legal sovereignty, giving them the ability to experiment with innovative regulatory systems much like China’s provinces.

 

What is a new non-AI technology that has played an important role in the development of your world?

Improved governance technology has helped societies to better navigate the “bulldozer vs vetocracy” axis of community decision-making processes. Using advanced coordination mechanisms like assurance contracts, and clever systems (like Glen Weyl’s “SALSA” proposal) for pricing externalities and public goods, it’s become easier for societies to flexibly make net-positive changes and fairly compensate anyone affected by downsides. This improved governance tech has made it easier to build lots of new infrastructure while minimizing disruption. Included in that new infrastructure is a LOT of new clean power.

Solar, geothermal, and fusion power provide most of humanity’s energy, and they do so at low prices thanks to scientific advances and economies of scale. Abundant energy enables all kinds of transformative conveniences:

  • Cheap desalinization changes the map, allowing farming and habitation of previously desolate desert areas. Whole downtown areas of desert cities can be covered with shade canopies and air-conditioned with power from nearby solar farms.
  • Carbon dioxide can be captured directly from the air at scale, making climate change a thing of the past.
  • Freed from the pressing need to economize on fuel, vehicles like airplanes, container ships, and self-driving cars can simply travel at higher speeds, getting people and goods to their destinations faster.
  • Indoor farming using artificial light becomes cheaper; instead of shipping fruit from the opposite hemisphere, people can enjoy local, fresh fruit year-round.

 

What’s been a notable trend in the way that people are finding fulfillment?

The world of 2045 is rich enough that people don’t have to work for a living — but it’s also one of the most exciting times in history, running a preposterously hot economy as the world is transformed by new technologies and new ways of organizing communities, so there’s a lot to do!

As a consequence, careers and hobbies exist on an unusual spectrum. On one end, people who want to be ambitious and help change the world can make their fortune by doing the all the pressing stuff that the world needs, like architecting new cities or designing next-generation fusion power plants.

With so much physical transformation unleashed, the world is heavily bottlenecked on logistics / commodities / construction. Teams of expert construction workers are literally flown around the world on private jets, using seamless translation to get up to speed with local planners and getting to work on what needs to be built using virtual-reality overlays of a construction site.

Most people don't want to hustle that much, and 2045's abundance means that increasing portions of the economy are devoted to just socializing and doing/creating fun stuff. Rather than tedious, now-automated jobs like "waiter" or "truck driver", many people get paid for essentially pursuing hobbies -- hosting social events of all kinds, entering competitions (like sailing or esports or describing hypothetical utopias), participating in local community governance, or using AI tools to make videos, art, games, & music.

Naturally, many people's lives are a mix of both worlds.

If you liked what you've read so far, remember to visit the official competition entry page to read the two day-in-the-life short stories and provide feedback!A Note of Caution

The goal of this worldbuilding competition was essentially to tell the most realistic possible story under a set of unrealistic constraints: that peace and prosperity will abound despite huge technological transformations and geopolitical shifts wrought by AI.

In my story, humanity lucks out and accidentally kick-starts a revolution in good governance via improved institution design – this in turn helps humanity make wise decisions and capably shepherd the safe creation of aligned AI.

But in the real world, I don’t think we’ll be so lucky.  Technical AI alignment, of course, is an incredibly difficult challenge – even for the cooperative, capable, utopian world I’ve imagined here, the odds might still be against them when it comes to designing “superintelligent” AI, on a short schedule, in a way that ends well for humanity.

Furthermore, while I think that a revolutionary improvement in governance institutions is indeed possible (it’s one of the things that makes me feel most hopeful about the future), in the real world I don’t think we can sit around and just wait for it to happen by itself.  Ideas like futarchy need support to persuade organizations, find winning use-cases, and scale up to have the necessary impact.

Nobody should hold up my story, or the other entries  in the FLI’s worldbuilding competition, as a reason to say “See, it’ll be fine – AI alignment will work itself out in the end, just like it says here!” 

Rather, my intent is to portray:

  • In 2045, an inspiring, utopian end state of prosperity, with humanity close to achieving a state of existential security.
  • From 2022-2044, my vision of what’s on the most-plausible critical path taking us from the civilization we live in today to the kind of civilization that can capably respond to the challenge of AI alignment, in a way that might be barely achievable if a lot of people put in a lot of effort.


Discuss

Why don't 4 lottery wins prove information from a higher power?

18 мая, 2022 - 18:10
Published on May 18, 2022 3:10 PM GMT

I would like to see a specific analysis of the errors, because I can only see for sure that frequency statistics are used, and not Bayesian. And so, the task: one popular Russian mathematician claims that since a woman won the lottery 4 times in a row, which is absolutely incredible, which is clear to any mathematician who, unlike mere mortals, sees the true essence of the universe, for example, that the chance of such a coincidence is one in a million billion billion, so this woman had information from the Christian god, because only he can break the laws of the highest of the sciences of mathematics, which is the language of god.



Discuss

Understanding Gato's Supervised Reinforcement Learning [Link Post]

18 мая, 2022 - 14:08
Published on May 18, 2022 11:08 AM GMT

The recent publication of Gato spurred a lot of discussion on wheter we may be witnessingth the first example of AGI. Regardless of this debate, Gato's makes use of recent development in reinforcement learning, that is using supervised learning on reinforcement learning trajectories by exploiting the ability of transformer architectures to proficiently handle sequential data.

Reading the comments it seems that this point created some confusion to readers not familiar with these techniques. Some time ago I wrote an introductory article to how transformers can be used in reinforcement learning which may be helpful to clarify some of these doubts: https://lorenzopieri.com/rl_transformers/ 



Discuss

Gato's Generalisation: Predictions and Experiments I'd Like to See

18 мая, 2022 - 10:15
Published on May 18, 2022 7:15 AM GMT

I'm deliberately inhabiting a devil's advocate mindset because that perspective seems to be missing from the conversations I've witnessed. My actual fully-reflective median takeaway might differ.

My covid has made writing difficult at the moment, and I haven't had the energy to gather citations or fully explain the detail for some of the assertions in this post.

I believe (for various reasons not detailed here) that qualitative advances in general agentic performance from artificial systems are likely this decade and next - I just don't think Gato represents progress in that direction. I'm not particularly surprised by anything in the Gato paper[1]. Naturally then, I'm against hyperbole around the capabilities demonstrated.

There is not enough information in the paper to say either way, but it may be the case that Gato represents a novel and cutting-edge distillation technique and if so, I would prefer if it were framed that way than as an increment towards generalisation ability of artificial agents!

The authors do a reasonable job of reporting their experiments (though I sense some motivated reasoning in the presentation of some results[2]), but the PR version of the story (at least as I have encountered it) departs from the mostly-honest paper itself[3].

I focus here almost entirely on sequential control, since that is the aspect of the discussion around the Gato paper which seems most clouded, while being at the centre of the purported generalisation, and in which I have the most interest and expertise.

Techniques and phenomena

We've known for many years that attention mechanisms (early 2010s) allow parameter sharing over sequences and over inputs of variable size/shape, that behavioural cloning (early 1990s?) often works with enough expert examples, that representation transfer (embeddings, early 2010s?) works, and that fine-tuning parameters (2000s? 2010s?) is faster than learning from scratch.

What should the key empirical takeaway be?

In my reading, if you subtract the already-known phenomena, and pass over the mostly-ignored distillation aspect, there is an important empirical finding in the Gato paper. I'm not sure if it's novel per se, but probably at this scale in this domain.

Namely, in our current systems, fine-tuning is typically faster when pretraining includes closely related tasks (e.g. DM Control Suite and Meta-World), slow when pretraining does not include closely related tasks (e.g. every control task without domain-relevant pretraining; including RGB stacking), and actively worse when pretraining includes distracting unrelated tasks (e.g. the single held-out Atari task, Atari boxing, is distracted by any pretraining in the reported experiments!).

Fine-tuning progress demonstrating benefit of closely related pretraining, and neutrality or harm of unrelated pretraining

Fine-tuning progress demonstrating benefit of closely related pretraining, and neutrality of unrelated pretraining

There is the additional novelty of goal-conditioning by context-prompting with a completed task, both during training and deployment (section 2 of the paper). Goal-conditioning is an interesting and relatively recent area of research. As with all such techniques I am aware of, this conditioning technique requires explicit presentation of a goal state which is structurally very similar to those seen in training. As briefly discussed in the paper, this could be applicable in the context of perceptual variation for closed-environment control (like some industrial robotics tasks) but this is fundamentally separate from general agency.

Comparison to other results regarding sequential control

Let's consider a comparison to the DQN papers published in NIPS and Nature in 2013.

Data efficiency

Note that the Gato control tasks were all learned by behavioural cloning on pretrained expert policies, each of which used a dedicated specific learning architecture, and were in many cases exposed to >10^8 environment steps for training each (depending on task; more on Atari; see appendix F of the paper).

Compare Nature DQN which had a quarter of the environment interaction experience per task on Atari.

Parameter count

Recall that the NIPS DQN paper learned Atari games without demonstrations (albeit per-game) using networks with less than a million parameters, and the final paper published in Nature used parameter counts little over a million. That's three orders of magnitude less than the larger Gato network (1.18B) and two-and-a-half less than the medium one used in ablations (364M), and this is before serious algorithmic and architectural optimisation investments started in deep RL - so we're not looking at a small network with surprising capabilities here, but rather a decently-sized network for the domains in question.

NIPS DQN[4]

type spec shape params 0 input 84x84 * 4 channels 84x84x4 - 1 conv 8x8 stride 4 * 16 channels 19x19x16 16*(8*8*4) = 2^12 2 conv 4x4 stride 2 * 32 channels 8x8x32 32*(4*4*16) = 2^13 3 ffn 256 256 256*(8*8*32) = 2^19 4 output <= 18 <= 18 <= 18*256 ~= 2^12

Nature DQN[5]

type spec shape params 0 input 84x84 * 4 channels 84x84x4 - 1 conv 8x8 stride 4 * 32 channels 19x19x32 32*(8*8*4) = 2^13 2 conv 4x4 stride 2 * 64 channels 8x8x64 64*(4*4*32) = 2^15 3 conv 3x3 stride 1 * 64 channels 6x6x64 64*(3*3*64) ~= 2^15 4 ffn 512 512 512*(6*6*64) ~= 2^20 5 output <= 18 <= 18 <= 18*512 ~= 2^13

(I assumed here they didn't use padding on the conv layers, and I ignored bias terms. The overal parameter count doesn't change much either way.)

Note also that the original DQN was able to achieve that performance while computing over only the 4 most recent frames, while the attention mechanism (and the recurrent mechanisms used in the expert policies) for the Gato paper will have access to much more history than that per decision step (perhaps ~50 frames for Gato, based on the context length of 1024 and image-tokenisation into 20 patches; see sections 2.3 and C.3).

Embeddings

In the Gato paper ResNet embeddings are also used for the image patches, but I can't find the details of these components specified - perhaps their parameters are in the low or fractional millions and either omitted from or absorbed into the parameter counts? But given that they're using residual connections, these image embeddings alone are presumably deeper than the DQN networks, if not larger.

Comparison conclusion

DQN used orders of magnitude fewer parameters, multiples less environment interaction experience, and orders of magnitude less compute per step, than Gato, on Atari.

Nature DQN Atari is a collection of neural networks. Gato's primary artefact is one (much much bigger) neural network (if you ignore the many individual neural networks used to learn the expert policies prior to behaviour cloning.)

Generalist or multi-purpose?

There are also learned embeddings for 'local position' (image-patch position and shape-relative embeddings for other input types). This is a sensible move and possibly the part of the paper which deserves to be called 'generalist' (as in, a general attention-based ML architecture).

In light of the above observations and comparisons, and some additional speculation detailed below, I consider the Gato artefacts ('agents') no more general than the collection of experts they were distilled from, though certainly they provide a multi-purpose model and demonstrate that distillation of multiple tasks into the same ML artefact is possible if you tag things appropriately and take care during training to avoid catastrophic forgetting.

Speculation

My best guess is that the domains and tasks trained have easily-detectable 'embedding signatures' via a combination of

  • explicit modality-dependent embedding (images vs text vs actions)
  • domain-dependent position encoding 'signatures' in the token-stream

Thus, I imagine as the Gato training algorithm learns it is doing something like

  1. learning good embeddings for different input modalities (as a consequence of diverse training data)
  2. extracting the 'embedding signatures' and internally tagging domains and tasks, for downstream 'switching'
  3. (most speculative) mostly routing domains and tasks to be learned/memorised 'separately'
  4. doing the usual attention-based sequence prediction thing on each domain/task

Of course it won't be as clean as this, but I bet a really good interpretabilitator would pull out some structure resembling the above pattern[6].

To the extent that Gato artefacts provide 'generalisation' capability, I expect it to be through transfer of good-quality representation embeddings learned over many related tasks, computed over the combination of the explicit token and position embeddings and the earlier layers of the attention network.

When domain- and task-specific feature representations are available, supervised fine-tuning on expert demonstrations should naturally be expected to rapidly and reliably recover good performance.

Any residual generalisation benefits from fine-tuning of cross-domain pretraining vs training-from-scratch may come from 'burn in' of the parameter space, effectively spending some training cycles getting it into a better 'initialisation' than the one that comes out of the box.

I do not expect any generalisation to come from anything resembling a general strategic, agentic, or planning core. I don't believe any of the Gato artefacts houses such a thing. For what it's worth, reading the paper, I get the impression that most of the authors also take this last position.

Experiments I'd like to see More hold-outs?

Purporting to demonstrate agentic generalisation demands more than four held-out tasks, and perhaps more than one held-out Atari task. Of course it costs money and time to run these experiments so they probably have not been tried yet.

Ideally there would also be a demonstration on actually out of distribution tasks. If there were any meaningful success here, my mind would be substantially changed, considering the reported negative generalisation performance on the single held-out Atari game.

Compute comparison between experts and distilled version

A comparison I would like to see would be between the Gato artefacts and the various expert policy models they were distilled from on

  • parameter count
  • runtime compute per step

If these comparisons are favourable, as noted above, it may be that Gato is best thought of as a novel technique for distilling policies and other expert behaviour, which would be an interesting result.

My best guess is that these comparisons are mostly not favourable to the Gato artefacts, but I'd be interested by counter evidence.

Ablation of image pretraining

For some tasks in the Gato paper, there was some residual benefit to pretraining on 'no control data' aka 'all but control data', compared to training from scratch. I imagine almost all of this benefit comes from pretraining the image embeddings. Note that the authors also speculate

agents in the DM Lab environment are fed images which, despite being simulated, are natural looking. Therefore, transfer from image captioning or visual grounded question answering tasks is possible.

This could be falsified by introducing new ablation categories for 'no image data' (include control but not image-based control) and 'no control or image data'[7]. In light of the above quote, I'm somewhat surprised not to find experiments like this in the paper.

Burn-in of parameter space

Initialisation of network weights is an area of experimentation and study; many approaches and heuristics exist, some of which will lend themselves to better learning rates than others.

A network at initialisation will have poorer weight/parameter settings than after some training(citation needed).

Here I'm referring to a fuzzy concept for which I do not have good technical language. Obviously parameters at initialisation are worse than after training (assuming training better than a random walk). But in addition to what we might think of as 'fine grained' learning and memorisation, there are also 'coarse grained' properties of initialisation which are poor, not specifically because they produce incorrect results, but because they are not yet in a good, readily-trainable region of parameter space.

For concreteness, consider properties like 'overall magnitudes' and 'diversity of weights' and the other heuristic properties which initialisation approaches are supposed to help with. The phenomenon of some amount of training cycles being spent 'burning in' the network is a common intuition in machine learning as practiced.

I don't have any experiment in mind for this one!

Perhaps ablating relative to amount of 'burn in' training on totally-unrelated data? But results of such experiments seem insufficiently-hypothesis-splitting to me: evidence for 'burn in' might also look like evidence for learning 'general transferrable agency'. I don't think the latter is what's happening, but the whole point of an experiment would be to disambiguate those hypotheses.

Uncovering implicit domain signatures

Evidence for 'domain signatures' or 'task signatures' would be the ability to 'easily' classify domain or task.

How many layers of the network's activations is required to classify domain/task by linear regression or small single-hidden-layer FFN?

  • taking padded single 'timestep frames' as input
  • taking n tokens as input
  • are the post-embedding raw tokens enough?
  • what about individual early layers of the attention network?
  • what about individual late layers of the attention network?
  • is there a network depth boundary where there's a phase transition?
  • how many principal components can we throw away and still classify?
  • how much distortion can we add and still classify? (e.g. squash activations to sign)
Explicit identification of domain- and task-tagging

Interpretability techniques applied to directions in the layer activations.

Can we uncover layer activation directions which correspond to domain- and task-tagging?

  • note that the tags and switching are likely to be rotated with respect to the canonical basis of the vector space of activations. See Features in Olah et al - Zoom In; Transformer attention networks in particular impose no real constraint on the orientation of features
    • so you might not find particular attention heads or weights dedicated to particular domains, but rather something more cryptic
  • can we locate domain- or task-specific highly-active directions?
    • what layers are they in?
    • is there a depth boundary where there's a phase transition?
Conclusion

There is a lot of great engineering reflected in the Gato paper, but the artefacts admit more potentially-revealing experiments than have yet been performed.

Depending on details not revealed in the paper, the techniques may represent a novel practical model-distillation technique.

I don't see evidence for the Gato artefacts being generalists (or agents, in a strict sense, for that matter), and my opinion is that claims to this effect are at best hyperbolic or confused. Readers (and writers) should take care to understand the multiple different possible connotations of words like 'general' and 'agent'.

  1. On the other hand, I do expect the discourse and framing to be such that the Gato paper is inevitably cited in any work from now on which does make progress in that direction (along with anything else which purports to). ↩︎

  2. Some examples I gathered on a subsequent reading:

    Section 4.1: (emphasis mine)

    For the most difficult task, called BossLevel, Gato scores 75%

    (footnote mentions that other tasks in fact scored lower)

    Section 5.2: (emphasis mine)

    In this section we want to answer the following question: Can our agent be used to solve a completely new task efficiently? For this reason, we held-out all data for four tasks from our pretraining set

    Section 5.5:

    The specialist Atari agent outperforms our generalist agent Gato, which achieved super-human performance on 23 games. It suggests that scaling Gato may result in even better performance.

    Appendix E:

    We evaluate agent every 100 learning steps. Each evaluation reports the average of 10 runs of a given checkpoint. The moving average of 5 such scores is computed (to gather 50 runs together). The final fine-tuning performance is defined as the maximum of these smoothed scores.

    ↩︎
  3. Based on professional experience, I react also (perhaps unreasonably) cynically to the density of phraseology using the 'generalist agent' bigram, and to the conflation of the techniques, training methods, architectures, and trained artefacts in the paper and public discussions under a single anthropomorphising moniker, 'Gato'.

    DeepMind publications usually read with much less of the marketing and hype aesthetic than this. ↩︎

  4. Mnih et al - Playing Atari with Deep Reinforcement Learning https://arxiv.org/pdf/1312.5602v1.pdf ↩︎

  5. Mnih et al - Human-level control through deep reinforcement learning https://www.datascienceassn.org/sites/default/files/Human-level Control Through Deep Reinforcement Learning.pdf ↩︎

  6. I suspect Transformer-style attention networks may be particularly well-suited to doing this sort of context-signature-tagging and switching-routing-separation and this is a large part of my model of how they work ↩︎

  7. Although see speculation on burn-in, which might confound such an experiment ↩︎



Discuss

I just watched the Open C3 Subcommittee Hearing on Unidentified Aerial Phenomena (UFOs). Here's a succinct summary and commentary + some background

18 мая, 2022 - 07:15
Published on May 18, 2022 4:15 AM GMT

Background on UFOs, Disclosure, and Rationality

There have been reports of various strange flying objects for a very very long time (read the paragraph with with "Boeotia". Note, a "hogshead" seems to be a type of barrel.). For me, it wasn't until quite recently that it became really unambiguously clear to me that something is really going on there. I'd recommend looking into the princeton/nimitz "tic-tac" incidents specifically. At least 6 navy staff on those boats have very publicly and extensively testified to having seen a very strange propulsion technology. I've come across no story as to how, or why any human faction would be keeping a propulsion technology like that secret, and out of deployment for so long.
(A half-baked theory though: Perhaps this propulsion tech could be used to make devilishly fast, non-interceptible ICBMs, the existence of which would make the world worse, and is therefore a technology that we should refuse to talk about for as long as possible?)

However, it's possible that it wasn't a propulsion technology, and that it's more of a plasma image projector used in consort with a radar spoofing technology, as futurist David Brin seems to confidently believe. In that case it's a lot easier to understand how the technology could have been kept hidden - it hasn't been mature for very long, and it has fairly limited applications, and it isn't salacious enough to leak about.
So this would be one of my dominant hypotheses, the tic-tacs really look like they're just this, to me. But there are a few contrary details in the account of commander Fravor and US officials swear up and down that they'd tell us if that were it.

So I kinda have to keep paying attention to this stuff.

But isn't it wildly implausible? How can a rationalist entertain the possibility that aliens would be like this, or behave like this? The econ/tech-eschatology of it doesn't make any sense!

I'm sympathetic to that. My intuition says that if aliens had reached us, their construction projects would have filled our night sky, they wouldn't be hiding from us, because whatever the value of hiding, the costs are greater. Robin Hanson talks extensively about the dynamics of economically plausible aliens and finds that it wouldn't generally support these sorts of observations.

But the observations seem to be disagreeing with my intuitions, so I seek a more detailed model.

Robin Hanson also talks extensively about the a less prevalent but still believable set of dynamics that we might expect to see in panspermia-sibling aliens who happen to have developed a sclerotic world government (a development which is not especially unlikely, and might happen to us) that could temporarily resist the laws of instrumental convergence towards "grabbyness".
That story would support these sorts of observations.

Regardless, if you are sure that it's definitely not aliens, you should be extremely interested in the possibility that humans, hence, appear to have created practical alcubierre drives.
I think that would warrant some discussion.

Summary and Commentary of the Proceedings of the Open Subcommittee

(The subcommittee hearing can be watched here)

There's a new US govt UFO investigation program: Airborne Object Identification and Management Synchronization Group, "Or, AIMSOG".

Congressman André Carson:

Today we will bring that organization out of the shadows. ... Unidentified Aerial Phenomena are a potential national security threat, and they need to be treated that way. For too long, the stigma associated with UAPs has gotten in the way of good intelligence analysis, pilots avoided reporting or were laughed at when they did, DoD officials relegated the issue to the back-room or swept it under the rug entirely, fearful of a skeptical national security community. Today, we know better. UAPs are unexplained, it's true, but they are real, they need to be investigated, and many threats they pose need to be mitigated.

  • Possibly new video of something reflective going past or being passed by a plane. Later in response to a clarifying question of Mr Himes, Mr Bray says yeah it's "probably moving very very slow".
  • They've figured out what that weird night vision footage of hovering triangles were! Just US drones producing a really weird camera lens effect in an IR lens being filmed by another camera. "We're now reasonably confident that these triangles correlate to unmanned aerial systems in the area. The triangular appearance, is a result of light passing through the night vision goggles, and then being recorded by an SLR camera"
  • (after faffing around trying and failing to get VLC to show a still of the flyby thing from that maybe new video. (It doesn't occur to anyone to try slowing down the playback :|||| like, should I be reading into the fact that there was this very basic technical puzzle in front of them and in all that time, not one of these guys thought of this))

    Mr Schiff: "Is this one of those situations where it was observed by the pilot and it was also recorded by the aircraft's instruments?", "Ah, we'll talk about the multisensor part, in a later session" (a closed one? :|| Why. It's possible this is just because talking about what the sensors saw would require talking about the sensors themselves, which is generally secret info.) "But in this case, we have, [gestures at the footage] at least that." :< neither confirming nor denying correlation from other sensors.
    I think I like Mr Schiff a lot. He is a live player. Regarding the hovering triangles, he questions this frankly surprising explanation optical artifacting, and asks whether they've reproduced that effect experimentally. Yes, apparently, it has been done.
  • Mr Wenstrup: 'have allies or adversaries reported similar sightings', "we should save that for a closed session" :[
  • Mr Gallagher asks them about the Malmstrom incident (I think this is it) where a UAP seemed to interfere some nuclear missiles. They say they haven't looked into it or heard of it. I'm as concerned as Gallagher is to hear that. They say they don't have the resources to follow up on that sort of story without an authoratative figure requesting it. Mr Gallagher says 'I don't claim to be an authoratative figure but for what it's worth I would like you to look into it'. Moultrie says "will do".
  • Mr Krishnamoorthi asks whether we have records of anything under the sea. Moultrie: "I think that would be more appropriately addressed in closed session, sir"
  • Mr Moultrie confirms that they do have processes "to ensure that we are not potentially reporting on something that may be a developmental platform or a US operational platform that is performing either testing or performing a mission".
    He states that there don't seem to have been any cases where they've accidentally reported a potential US technology. I don't place much stock in this, though. They wouldn't be able to do this job at all if they weren't willing to lie about whether something is a US technology, alas, we should expect them to lie about those cases, so maybe it doesn't ultimately mean very much that the government are consistently denying having any tech programs that could explain the observations.

I approach a model: It's conceivable that that what's actually going on here is, the story of Alien UFOs is useful for hiding novel technologies, but it's also been an impediment to national security, it's created a stigma around reporting, which means that if a rival nation actually had propulsion technologies like this, their radar crews might not report it. The US wants to maintain the cover of the AUFO story, but they also want to dissolve the stigma. I think this new group are going to do that exceptionally well.



Discuss

Workday Air Quality Measurements

18 мая, 2022 - 05:10
Published on May 18, 2022 2:10 AM GMT

I continue to be very excited about my new M2000 meter, and I brought it with me today to measure air quality during my commute and workday. Here's what I found:


full-size image

Time Activity CO2 (ppm) pm2.5 ug/m3 8:00 Left house 426 2.9 8:15 Davis station 468 69.9 8:24 Red line, new car 802 29.3 8:36 Kendall station, outside 477 11 8:40 Hallways, elevators, breakfast in cafe 563 2.8 9:18 Office, door open 475 1 10:00 Closed door 514 1 10:31 Open door 553 1 11:02 Team meeting, 14 people, ~15x30 1052 1 12:04 Hallways, elevator 686 1.6 12:10 Lunch in cafe 943 2.6 12:50 Hallways, elevator 763 2.1 12:57 Office, door open 537 1 2:02 Closed door 539 1 2:31 Two people 599 1 2:54 One person, open door 587 1 3:01 Closed door, two people 598 1 3:31 Open door, zero people 538 1 4:04 Closed door, two people 591 1 4:38 Kendall station 487 36.6 4:44 Red line, old car 953 21.6 4:56 Davis station 537 134.5 4:58 Outside 445 2.25

I spent most of my workday in my office, with the door open. I closed the door for meetings, either virtual or in person. The "zero people" at 3:31 is me going on a walking 1:1: with a report and leaving my meter behind at my desk.

Currently I wear a mask on the subway (and stations) and at work when I'm not in my office or a conference room with a small number of people. Based on these readings, it would be safe from a covid perspective to remove my mask in the subway station, but given the high level of particulate pollution I might as well leave it on.

(At this point I'm not wearing a mask because I think it would be directly harmful to myself or my family to get covid, but instead because the steps we would need to take to reduce the risk of infecting others once we knew we were infected would be very inconvenient.)



Discuss

Страницы