Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 7 минут 30 секунд назад

Starting too many projects, finishing none

5 мая, 2022 - 18:23
Published on May 5, 2022 3:23 PM GMT

Lately I have been starting a ton of projects, but dropping them once I lose interest. I come up with a cool paper idea, but next week I have new idea I'm excited about. Almost every idea I have requires extensive commitment to see return, so the short projects are wasted.

Does anyone else have this experience? What strategies did you use to pick the high expected-value projects and stay with them?



Discuss

Repeal the Foreign Dredge Act of 1906

5 мая, 2022 - 18:20
Published on May 5, 2022 3:20 PM GMT

There are a lot of ludicrously terrible government laws, regulations and policies across all the domains of life. My Covid posts have covered quite a lot of them.

Yet if I had to pick one policy that was the Platonic ideal of stupid, the thing that has almost zero upside and also has the best ratio of ‘amount of damage this is doing to America’ versus ‘reasons why we can’t stop being idiots about this’ there is (so far) a clear winner.

We must repeal the Foreign Dredge Act of 1906. It says, to paraphrase, no underwater digging – to repair ports, or build bigger ones, or fix waterways – unless the boat doing the digging was built in the US, and is owned and operated by Americans. (This isn’t about shipping – that’s the Jones Act, which has similar ownership rules for shipping within the US, and which we’ll get to later.)

I claim that, EA style, this is highly (1) important, (2) tractable and (3) neglected.

There’s a bunch of talk recently about the Dredge Act which is how I noticed it, but that’s different from the actions that actually lead to repeal – it’s still neglected. An illustration of this is that my exploration of this led to it having a Wikipedia page. Until May 2nd, it didn’t.

The actions that could repeal the act mostly involve a relatively small amount of standard-issue lobbying effort – so it’s tractable.

Given how much it could do for our ports and thus our economy, as well as the reclamation projects we could do, repeal seems pretty damn important.

The goal of this post is to explain what the hell is going on here and defend those three claims.

Odd Lots

This topic was entirely off my radar screen until I listened to a recent episode of one of my favorite podcasts (transcript here): Odd Lots. Odd Lots is hosted by Joe Weisenthal and Tracy Alloway. If you are at all into economics or economic-style thinking, this podcast is for you. Often they tackle questions of trading and market structure and interest rates, or the world of crypto, but they are at their best when they are asking about real world logistics and how that fits into the economic picture. Odd Lots is great most of all because it is centered in a profound curiosity about the gears of the system of the world.

Anyway, it all started when Tracy Alloway’s shipping woes (she’d been trying as an experiment to get a spot on a container ship crossing the pacific for months without success, which was very enlightening on what’s going wrong with shipping) took a turn for the personal, and her belongings got stuck on the ship Ever Forward in Chesapeake Bay. Which we struggled to get free because America lacks proper dredges, which led to a whole episode about dredging.

I’ll quote from it a bit, but I recommend listening to the episode directly.

What Is Dredging and What is it For?

From the official source:

Dredging is the removal of sediments and debris from the bottom of lakes, rivers, harbors, and other water bodies. It is a routine necessity in waterways around the world because sedimentation—the natural process of sand and silt washing downstream—gradually fills channels and harbors.

Dredging often is focused on maintaining or increasing the depth of navigation channels, anchorages, or berthing areas to ensure the safe passage of boats and ships. Vessels require a certain amount of water in order to float and not touch bottom. This water depth continues to increase over time as larger and larger ships are deployed. Since massive ships carry the bulk of the goods imported into the country, dredging plays a vital role in the nation’s economy.

Indeed.

There is also environmental dredging to eliminate contaminants. It can be used for land reclamation projects (like potentially expanding Manhattan) or building sea barriers. Dredging is used to free boats that get stuck (like the Ever Given or Ever Forward) or free up navigation on waterways in emergencies (like the Mississippi after Katrina).

Dredging is a bottleneck to the expansion and maintenance of ports, and in the resolution of emergencies. We can’t ship things if the boats can’t get in. The tasks cost relatively little money to do when done with the right tools, but solving these bottlenecks provides tons of marginal value compared to not solving the bottlenecks.

The entire supply chain depends on having working ports. Dredging companies and workers only capture a small fraction of the resulting consumer surplus.

What’s Wrong With American Dredges?

Their capacity levels suck. Here’s Tracy Alloway on Odd Lots:

And our previous guest who was talking about this, Sal Mercogliano, again, he has a great YouTube channel if you’re interested in what’s going on with the Ever Forward, but he was saying that the dredges that are on the scene of the Ever Forward right now can move about 60 cubic yards of mud in each, you know, every time they sort of dredge the bottom, whereas other types of dredges, international dredges, the kind that they had on scene with the Ever Given when it was stuck in the Suez Canal, those can move 70,000 cubic yards of material in one hour. So that gives you an insight into the different levels of dredging capacity we’re talking about.

America has none of the top 30 dredges in the world. Of the top 50 dredges in the world, America has three. The American dredges simply don’t have the same kind of level of capacity.

The result is that everything is slower and more expensive, when it can be done at all, and also the dredges that do exist are often taken away to work on something deemed higher priority so long-term projects get delayed indefinitely.

The dredges we would use are owned by Belgian and Dutch firms, that already have American subsidiaries that do work here and have contracts with our unions, but they can’t dredge.

They just can’t alongside dig in the sand because of this 1906 law. And it costs America millions or tens of million of dollars of jobs and billions of dollars. If you are in Savannah, you spent over a billion dollars for a port deepening project that would’ve cost under $500 million. And if you are in Virginia right now, you are spending, it was supposed to be $350 million. It’s now $450 million for a project that should cost hundreds of millions less.

Why Can’t We Build Good Dredges?

I mean, we could, in theory and if we were willing to pay enough and wait for many years, but we don’t. The explanation given on Odd Lots is that the American market isn’t big enough, and we’re the only country other than China with restrictions on who can dredge.

So the U.S. dredging market right now, maybe it’s a billion dollars, maybe with coastal protection becoming more urgent and, you know, even beach replenishment becoming a much more kind of an every year thing if you’re gonna save your tourist season in North Carolina, the market’s maybe more than a billion, but it’s not a huge market. And in fact, the global dredging market is probably about $20 billion.

The story here is that American construction costs are expensive because there are large fixed costs involved. The existing companies have a comfortable oligopoly in this small market, and not enough incentive to go big and build huge top-of-the-line dredges, so the fixed costs don’t get paid. It’s not obvious whether or not we even have the logistical capacity to build on par with the best such ships out there, but it is clear that there is no (non-regulatory) reason for us to have that capacity, and that any such building effort would in the best case take many years even if everything went perfectly.

The future largest American dredging ship is currently under construction.

At 15,000 cubic yards, the dredge—designed in collaboration with Hockema Whalen Myers Associates Inc. (also of Seattle)—has a length of 420 feet, a breadth of 81 feet and a draft of 28.5 feet.

While the dredge won’t be completed until 2023, it was able to achieve funding by a U.S. bank-led syndication. Schorr says the total cost of the vessel will be over $100 million once completed.

That’s far from nothing, but it is not going to rival the top European ships in terms of size or capabilities.

Could this problem be solved by simply commissioning world class dredges here in America, even if that cost more money than building them elsewhere?

This podcast from 2018 is about shipbuilding in general but points to a lot of the excuses that people make for why American shipyards aren’t competitive.

This paper compares American construction costs to foreign construction costs for different kinds of ships, although it doesn’t consider dredges.

If we presume that American contracts currently pay roughly double the price for the same dredging work, and that the dredging market overseas is competitive (which by all reports it is), and that costs would be relatively additionally high here, then this implies the venture of ‘build a world class dredge here in America’ would be unlikely to be profitable.

That goes double given the uncertainty. If at any time the Dredge Act gets repealed, you could suddenly have a $200 million ship that you paid $600 million to construct.

I don’t blame the American dredging companies for not being eager to invest in lots of extra capacity with that hanging over their heads. To be a worthwhile business under those conditions means making unusually high profit margins while controlling your risk. Also it’s an oligopoly. Which all in turn, for the country, means very expensive dredging and not that much of it.

Jones Act Problems

Remember the Jones Act? The Jones Act says that if you engage in shipping between two American ports, you can only do so in an American built ship, with an American crew, flying the American flag.

When I said ‘of course we should repeal the Jones Act’ several people said no, the Jones Act has a good reason behind it. And that purpose is to ensure an American merchant marine that could be commandeered in time of war.

It is expensive to fly under the American flag, it is expensive to use an American crew, and American shipyards are completely uncompetitive, so the result of this act is that we mostly stopped shipping things between American ports.

Which of course means you also don’t get the intended merchant marine fleet. A lesser requirement that made the ships useful in war, without imposing additional requirements like making the ship in America, would at least do something useful.

Thus the Jones Act is rather terrible, but it is perhaps not as impactful as it sounds. Our geography is such that we mostly lose little by imposing a soft ban on shipping between American ports. It certainly doesn’t help, but I’ve been persuaded pending further investigation that it’s not the biggest deal.

This is relevant to the Dredge Act because a dredge has been ruled to be a Jones Act vessel. Thus, if the Dredge Act was out of the way, the Jones Act would still impose the same effective requirements.

Lobbyists defending the Dredge Act are using this to claim that repealing the Dredge Act means also repealing the Jones Act, which they say would be terrible. Thing is, they are simply lying about this, as none of the bills introduced to repeal the Dredge Act touch the Jones Act. The actual solution in all such bills is to define dredging as not being shipping, leaving the Jones Act for another day.

Union Problems

It seems the other way the dredging companies are defending the Dredge Act is by convincing the unions to be afraid.

It’s opposition, a hundred percent because they make two arguments, okay, that this is gonna repeal the Jones Act. We’ve already addressed that, it has nothing to do with the transportation sector. It’s the construction sector, and they threaten the unions that these companies will come in. They’ll do the port of Houston and Corpus, then they’ll leave and then you’ll be without us, the American dredging companies. But in fact, we now know that there will be offshore windmill projects at least through 2040, 2050. So these companies have become big U.S. subsidiaries with U.S. offices, U.S. labor agreements. Of the 5,000 people you said are in the industry, almost all continue to work on the same exact projects. If the end of the Virginia project were open bid and that last $70 million were bid for$ 30 million, not $70 million, for example. And we saved $40 million in Virginia, the same people would do the job. It’s the same labor agreement, the same unions. It would just be on a vessel that was much more efficient for it.

There might not be American dredging companies anymore because those companies don’t offer a competitive product, but their replacements would be employing the same people. Yes, they’d work faster, a classic threat to jobs everywhere, but they’d also have more capacity and make it worthwhile to do more work, which should more than make up for that problem.

On top of that, other unions greatly benefit from having expanded and better working ports. If we also start doing reclamation projects, the possibilities scale rapidly.

Can you imagine if Manhattan had 15% more real estate to have commerce on, what that would be? Just the World Trade Center rebuilding was a massive boom for the construction unions and for New York. Think about that at 15% of new Manhattan, what that would be valued? That project is imminently doable.

And it’s not like Belgium and the Netherlands are hotbeds of anti-union activity.

So unions, collectively, should be actively in favor.

Hell, if it makes everyone involved feel better we can require by law that only unionized employees be allowed to dredge, it’s a fully unionized industry anyway so the law would be dumb but have almost zero practical effect.

Which leaves only the actual special interest, the American dredging companies sitting around collecting oligopoly protectionist rents by imposing orders of magnitude higher costs on the rest of us – and, of course, limiting international shipping by constraining capacity.

How Big is the Special Interest?

There are about 1,650 American dredge operators. As we noted above, those union jobs aren’t going anywhere, they’d simply get more done by having better tools.

In theory there are those who work in the shipyards that manufacture the dredges themselves, but there would be so much additional shipyard work from all the additional shipping, and the need to service the new dredges, that such workers need not be concerned. Busier ports are a win for everyone involved.

The primary players who lose are only the few existing American dredging companies. I didn’t put that much effort towards trying to find the combined market cap, but we can guess given they have 1,650 combined workers operating the machines. As an opponent, they seem eminently beatable, and as a loss they seem trivial. If their owners have diversified portfolios they shouldn’t even care at all.

But What About the Environment?

One possible counterargument is that we shouldn’t make it possible for us to dredge because dredging is bad, actually, as it ‘damages the environment.’ So by that logic we should be happy that we have made such activities much more difficult.

The first thing to note is that requiring us to use American dredges is very very bad in terms of the environmental impact of any given project, on two fronts. From the Odd Lots podcast: 

If you look at, actually at the modern dredges that are being built in European shipyards that are being used around the world, unfortunately just not in the United States, you see a couple of differences that actually make them more environmentally friendly. The first is that the newest and most modern dredges are using LNG as opposed to marine diesels. So they’re emitting a lot less emissions as they’re working. The second issue — and this was a real tragedy in Miami — is because the dredges that we use are so-called cutter dredges, that they weren’t powerful enough to chew basically through some of the rock that they needed to remove in order to create a deeper channel for cruise ships. They had to use blasting.

Blasting in turn causes a whole lot of unnecessary additional damage, for details see the transcript. If we are going to dredge, which to a large extent we are going to do no matter what, we should do it in a way that causes less damage – the same way that we should do it faster and cheaper and better. 

That doesn’t rule out a position of roughly ‘yes this is a no-good-very-bad way of limiting how much we dredge but it does limit it and that is what matters.’

I don’t know how to engage with that perspective as anything but opposition to civilization. If you don’t think we should maintain or create ports, make it possible to navigate rivers or free ships that get stuck – which are the primary reasons people dredge – then that’s not compatible with having a technological civilization. Perhaps there are other ways to work around that and still have a technological civilization, but they are orders of magnitude worse in terms of their consequences for the Earth.  

So yes, if you are opposed to civilization and progress and humanity’s prosperity and survival, then I suppose you should be in favor of keeping the Dredge Act of 1906. Fair enough.

How Bad Is The Dredge Act of 1906? Is it Impactful?

Seems pretty bad.

From the new Wikipedia article (and thanks for that, AllAmericanBreakfast):

Two countries, the United States and China, prohibit foreign dredging, and 15% of countries surveyed by the Transportation Institute have restrictions on dredging.[7] The U.S. Army Corps of Engineers and Government Accountability Office state that lack of dredging capacity and high costs are the cause of a 15-year delay in dredging the 10 most important US ports to accommodate post-Panamax depths. 90% of global dredging contracts are currently won by one of four Belgian and Dutch dredging companies Jan De Nul, Van Oord, Boskalis, and DEME.[8]

That confirms that we’re falling behind, but doesn’t give a sense of the magnitude of the damage.

This is Houston, from the Odd Lots transcript:

So now the greatest country in the world has a law preventing container ships from entering one of its greatest ports because they cannot get them in. So if we just dredged Houston at half the cost in a third of the time that would create and support over 1.6 million new American jobs, by lowering the cost of exports by over 15%, it would change our energy security picture.

Many tasks could be done much faster and cheaper with foreign dredges. There are many tasks our available dredges cannot do at all, including keeping major ports like Houston fully operational, or expanding our ports so they can accommodate larger and more economical modern ships.

It’s traditional to claim numbers like ‘1.6 million jobs,’ which I’ve seen attached to both Houston on Odd Lots or to collectively expanding all the ports, but the effect of expanding ports and other such infrastructure is cumulative over time in a way that makes any given number wrong. If you have to give a guess of this kind, it seems… kind of reasonable, actually. Being able to take your spices from one port and efficiently ship them to another port is both the best thing and also key to economic success.

Our lack of port capacity is a key bottleneck in our supply chain. I don’t know how much it has been contributing to inflation numbers, but I expect it to be substantial, as many of our goods get shipped here and the cost of that shipping has skyrocketed in both money and time. My mean guess is an effect here of several percent. Left alone this is likely only going to get worse.

More than that, I’m guessing this is a substantial permanent hit to trendline real economic growth while it persists. That is, it reduces growth in ways that compound over time. That’s the biggest game of all.

We can also add the problem where we cannot deal well with emergency situations and that this is also very expensive.

Here’s a concrete example from this post of how we can’t get our act together on this even for a true emergency, costing us at least billions.

At this time I was sent to the U.S. as a consultant for Hochtief Dredges from Germany. We had two large cutter suction dredges just finishing off a dredge in the mouth of the Orinoco River in Venezuela.

I went to the Army Corps of Engineers and told them I could have two, world-class capital dredges in New Orleans in less than three days. We reckoned we could cut a channel in the Mississippi in less than ten days. They were very excited. We met with several representatives from the ports and they were enthusiastic as well. The U.S. was losing hundreds of millions of dollars a day in the blockage.

They called a meeting along with Congress members from the area. I was then told that we couldn’t bring in our dredges to open the river because they were foreign dredges, run by a foreign company. The Corps of Engineers and some Louisiana politicians said they would try to get an exemption based on a national emergency. Unfortunately, the politicians concluded that they couldn’t make an exception for something as sensitive as the Jones Act. They eventually found a company called Great Lakes Dredges that had a vessel with proper, foreign, equipment on it installed on a U.S. bottom. But it took months to clear the Mississippi. We could have done it for twenty percent of the price they paid, and in ten days.

Speaking of the Mississippi River, it looks like dredging the lower Mississippi would be quite profitable as well, although this is one we are capable of doing now:

At ports along the mouth of the Mississippi, most ships loading soybeans can carry a maximum of 2.4 million bushels, and any additional weight in the hold puts the vessels in danger of scraping the riverbed. However, a mere extra 5′ in depth allows a ship to squeeze in 2.9 million bushels, at a small increase in transport costs. 

Translation: Digging the depth of the lower Mississippi from 45′ to 50′ could generate $461 million annually for the U.S. soybean industry — independent of supply and demand.

That’s the payoff confined to going from 45’ to 50’ and also confined to only soybeans.

Started in 2020, and scheduled for completion by 2022, the Mississippi River Ship Channel Dredging Project will cost roughly $270 million, and is expected to return $7.20 for every $1 spent, according to Corps of Engineers estimates. 

This seems like it has >100% ROI per year on soybeans alone, so that 720% return feels rather very low. So then consider what it would get us if we had unlimited capacity and cut our costs in half, and then dredged the entire Mississippi properly. Then apply that to all the other rivers and also the ports.

This image (from 2018) makes clear the extent to which our ports simply can’t handle modern ships due to failure to dredge.

A counterargument could be that even if we could do such projects in theory, perhaps we still wouldn’t in practice for other reasons like requiring approval from the Army Corps of Engineers and the associated environmental reviews?

At Congressional hearings the question was asked, “How long does it take to get full approval for a dredging project?” The answer was astonishing. The lead time for originating a dredging project, and the day when dredging started was sixteen years.

The post quoted in this section agrees that the Dredge Act is a bigger offender than the Jones Act, but still thinks the Jones Act matters as well. Whereas I got a decent amount of pushback from smart people on the Jones Act in terms of the size of its impact in practice – yes it kind of shuts down shipping between two American ports but it’s not clear how much that matters.

First Best Solution

Senator Mike Lee proposes the DEEP Act and as backups also offers three other bills. The DEEP Act seems like an excellent solution.

Here’s from the one pager (the full text is here):

Bill Specifics: The Dredging to Ensure the Empowerment of Ports (DEEP) Act would support more economic opportunities at our ports.

It would:

Repeal the Foreign Dredge Act of 1906

Require the Army Corps to create a new Nationwide Permit (NWP) for dredging projects at a port or the navigation channel of a port with clear regional conditions.

Require the NWP be issued for 10 years

Require the NEPA process for the NWP be completed within 2 years with only technologically and economically feasible alternatives considered

Require the Army Corps to eliminate the duplication between the Section 404 and Section 408 processes of the Clean Water Act

Remove EPA’s enforcement and oversight over the Section 404 permitting process under the NWP

Provide clear response times from the Army Corps for individuals seeking pre-construction approval for a dredging project so that project managers have certainty about the decision-making process.

Require any dredging project mitigation required by the Army Corps be technologically and economically feasible and within its jurisdiction.

This all seems excellent. The purpose of the last rule is non-obvious, but I believe it is to ensure that the EPA and/or state governments can’t claim jurisdiction and use that to delay projects. 

Not only does this repeal the Dredge Act, it also gets rid of a lot of other barriers to getting our dredge on within reasonable time. I’m especially excited by the NEPA provision.

I read the bill, and it reads like it was written by someone trying to get a port dredged. Who has experience with projects that couldn’t get the required approvals and paperwork and lawsuits handled, and Has Thoughts about how to fix that. I approve.

As a backup plan, the Port Modernization and Supply Chain Protection Act would repeal the Dredge Act but not do the other neat stuff.

As a further backup plan, the Allied Partnership and Port Modernization Act would allow NATO vessels to be used.

As a further backup plan, he also introduced the Incentivizing the Expansion of U.S. Ports Act, which modifies the Dredge Act to allow foreign-built vessels so long as America buys, flags and crews them. American union crews are going to be working the jobs anyway, so this would mean creating some sort of company to take possession (temporary or otherwise) of the dredge and flag it as American. That’s not great, but I’m guessing we could make it work in a pinch.

Lee has also introduced legislation to repeal the Jones Act, of course.

Second Best Solution

This post is amusingly titled “To New Critics of the Foreign Dredge Act: Welcome Aboard”, and includes several additional links to learn more. It suggests we might pass the Ship It Act (full text) rather than do an outright repeal, same as Senator Lee’s third proposal. I checked, and it turns out Lee introduced the bill in the senate in addition to the other four.

I read the bill in question, and I am certainly in favor of passing the Ship It Act. The non-dredging provisions are all about providing waivers of various requirements under the right circumstances. The dredging section doesn’t outright repeal the Dredge Act, but it does expand the list of allowed dredges to include anyone in NATO, which includes Belgium and The Netherlands, which have everything we need.

The whole bill reads as a compromise between the obviously correct action (repeal regulations that are getting in the way and that serve no useful purpose beyond a little narrow rent seeking at most) and an attempt to overcome motivated or dumb political objections by requiring waivers, keeping versions of many of the restrictions in place (e.g. NATO ships instead of USA ships) and phrasing the situation as temporary.

Is that a smart method of compromise? That’s an interesting question.

By structuring things around waivers, we’re digging the paperwork and complexity holes deeper rather than trying to climb our way out of the hole. In the long term, the cumulative weight of such things adds up. One can hope that once the waivers don’t cause any problems, they would turn into a formality and maybe eventually a full cancellation of the requirements, but I am skeptical. In the short term, it’s a lot of much-needed relief, and gets you most of the way there.

For dredging it gets you all of the way there, since the dredges we want to hire would be allowed, and as long as some worthy dredges are allowed it doesn’t matter that much if some others are excluded. It’s annoying but tasks can be shuffled around to make it work.

This also lacks some other very good provisions in the DEEP Act, which effectively likely would mean that dredging projects would remain very slow to happen. Unlike DEEP, this reads like it was written by someone who did not draw upon frustrating experiences trying to get projects to happen, and instead wants to create a path whereby projects might in theory happen at all.

Third Best Solution

This post echoes the claim on Odd Lots that our inability to fix our ports is costing us 1.6 million jobs, and also that we need a project to protect Manhattan from flooding (ideally by building more land) which can’t be done with the domestic dredging fleet but could easily be done with foreign dredges, and points out the plan is backed by the majority leader Chuck Schumer. It estimates our direct cost savings at $2 billion, although it’s not clear what time frame that covers.

The bill proposed there is even more of a kludge than the Ship It Act, where first you let American companies bid and then once they fail you can then let Europeans bid and if they win by enough you can give them the contract – again, jumping through hoops permanently in order to ‘prove’ what everyone already knows, that America can’t do this, as opposed to wanting to actually get the job done and fix America’s ports.

Other Simple Supporting Arguments

Wall Street Journal: Protecting US Dredges Kills Jobs.

A simple version of the argument.

What Now?

The first step was noticing the problem, and realizing this was indeed very low-hanging fruit. The second step is making others aware of the problem. The third step is actually working to get the law repealed.

In addition to writing this up, I have talked directly to a few different sources that have the potential to assist with the effort to repeal the Foreign Dredge Act. Some good questions have been asked. So far everyone seems to broadly agree on the opportunity – the whole point of picking this target is that it is not only a big win but the lack of an appreciable downside.

My model of why this hasn’t gotten done is that the benefits are sufficiently diffuse and/or their scope was sufficiently non-obvious or would take too long to be realized, or similar considerations, such that no one put in sufficient amounts of political capital and money to make it happen. It wasn’t enough of a priority.

My hope is that this can also constitute a sort of dry run on several fronts. Experience can be gained, relationships can be built, and it is an existence proof of the bills on the sidewalk that one can pick up and that are sufficiently high denomination to justify the effort. It’s also a proof-of-concept for various groups to actually fix things that we identify as broken.

Going into more detail would be beyond scope for now, but I think a lot of things get steadily easier as times get better, and all fronts help all fronts including everything from finding ways to build more houses in places people want to live to esoteric problems like pandemic preparedness. Bad times create zero-sum thinking.

Is This All Worth It?

For those who are inclined to consider all such things as potential ‘cause areas’ and are generally dismissive of progress studies, does this pass muster? As far as I can tell that should come down, from their perspective, to the numbers. How do you calculate how much something like this is worth, and how much does the effort cost per extra repeal of the act you achieve?

The cost per additional success is hard to know, but seems like it is in the mid-7s to low-8s in terms of digit range.

The direct benefits then need to be estimated.

The direct cost savings (as in, if we did the current set of jobs cheaper and faster) depends on the current size of the market. If we take the 5% at face value and the 11 billion worldwide size estimate here, and assume roughly 50% cost savings, we get $250 million/year. At a 5% discount rate we can value that at about $5 billion, plus the benefits of getting projects done faster, and doing more projects. Already this seems to be approaching the 1000:1 ratio where economic interventions make sense, but the real benefits are in what you do with the jobs you wouldn’t have otherwise done.

If the estimate of 1.6 million jobs checks out, we are already talking about single digit costs per job created, which should already compare favorably with third-world interventions even without any of the additional indirect benefits, of which there are many. The impact on inflation could be substantial even within a few years. 



Discuss

Covid 5/5/22: A Lack of Care

5 мая, 2022 - 18:10
Published on May 5, 2022 3:10 PM GMT

China cares a lot about preventing Covid.

I haven’t written an additional China post because my sources have not turned up much additional information, and the situation does not seem to have dramatically changed, so I’m waiting until situation warrants an update. One development was mass testing going on in Beijing, raising worries about lockdown there which could be important politically, but so far the lockdowns haven’t happened.

America does not care much about preventing or treating Covid.

We don’t care about buying or distributing Paxlovid. We don’t care about updating our vaccines. We don’t care about much of anything else either. Nor does the public much care about any of this either. Given the physical situation and what state capacity allows us in terms of alternatives, I am not even sure I would prefer things be a different way. Yes, it means among other things that we literally have a cure for Covid and are barely using it, but it does mean we don’t suffer from lots of extra prevention costs.

The good news is that most of us can safely ignore the whole thing and get on with our lives. Given the alternatives, ‘government does literal nothing’ is not obviously bad news. If they’d done literal nothing from the start we could well be in a much better spot. Alas, this literal nothing does involve things like preventing children from being vaccinated.

The Current Thing not only is no longer Covid, it seems that the invasion of Ukraine has also been replaced due to the leaking of a Supreme Court draft opinion on abortion. Unlike Ukraine, that does not seem like a situation in which my analysis would be news you could use, and I hope to avoid writing much of anything about it.

Executive Summary
  1. New subvariants of Omicron that spread faster are taking over.
  2. They are not deadlier.
  3. Our government is acting as if it does not care about Covid at all.

Let’s run the numbers.

The Numbers Predictions

Prediction from last week: 400,000 cases (+22%) and 2,720 deaths (+10%?)

Results: 358,439 cases (+9%) and 2,234 deaths (-10%)

Prediction for next week: 420,000 cases (+15%) and 2,275 deaths (+2%).

North Carolina reported 1,172 deaths yesterday, which is obviously a backfill. 1,146 of these were due to updated reporting, and I’ve removed them. With those gone, the number of deaths continues to decline even after the Easter weekend, and this drop is definitely genuine. I’m guessing the numbers that are getting reported are now reasonably decoupled from ‘from Covid’ deaths actually caused by Covid. I’d think they’d go up a little either way, but I wouldn’t have expected a drop this week.

On cases this was overall a good number but the increase in New York in particular is disappointing because it shows that we don’t have a clear peak waiting for us in the future. I’m going to predict a somewhat faster rise this week because I doubt the Midwest drop will get sustained.

Deaths

The deaths number going up this much shows that my prediction the previous week was indeed far too high, despite this coming in substantially higher than my median guess, confirming that last week was a cross between slower real growth than expected and the Easter holiday. This week had a huge jump in the South region.

Cases

BA.1,2,3,4,5

BA.1 gave way to BA.2. Now BA.2 is giving way to BA.2.1.12.

And so it goes, sub-variant gives way to sub-variant. There is no sign that BA.2.1.12 differs substantially in terms of case outcomes from BA.2 or BA.1.

Next up are BA.4 and BA.5 (Flashback to the movie Terminal Velocity: ‘What happened to three?’).

Here is a thread on a new study of cross-immunity between variants (data source).

The news isn’t great. There is reason to think that BA.4/5 might be better able to re-infect people, especially those who were not vaccinated, and thus could cause an additional wave. However they still respond well

Scott Gottlieb reaches similar conclusions.

Bloom Lab takes the same results, and tackles the exact mutations in BA.4/5.

This has all largely been the pattern. New variants make it easier to get infected despite vaccination or previous infection, but protection against severe disease and death remains mostly robust. As a result, additional waves are possible, but they do not case as much proportionate severe disease or death, and the wise move is largely to ignore the wave and go about one’s life. The bigger danger would be if we were unable to do that, but I am not much worried about that at the moment. That could be a big problem if physical circumstances got bad enough, but for now it is saving us.

Bloom’s call for updating the vaccines seems important, but the FDA disagrees. As the prevention section notes, they are dragging their feet and delaying updating into late fall for a variant we knew about last year. Utter disaster.

Physical World Modeling

Bill Gates, always helpful, is here to warn us that the worst of the pandemic may still be ahead.

“We’re still at risk of this pandemic generating a variant that would be even more transmissive and even more fatal,” the billionaire Microsoft co-founder and public health advocate told the Financial Times on Sunday. “It’s not likely, I don’t want to be a voice of doom and gloom, but it’s way above a 5% risk that this pandemic, we haven’t even seen the worst of it.”

This is a pretty weird hybrid of probability (great!) and not probability (less great?), what is ‘way above 5%’? My instinctive interpretation of this is something like ‘I would bet on this at 10% and my real odds are somewhat higher than that’ or something, so real odds in Gates’ mind of maybe 15%-20%, but I’d accept numbers as low as straight 10%. Chances are he doesn’t have a conscious probability estimate here, it’s more that he feels it’s definitely above 5%.

Gates is not reported as having presented evidence for this claim. Does it seem right? Purely in terms of deaths, I can’t disagree simply because 5% is not a lot and it seems fair to put this at more like 10%, and I wouldn’t have a strong disagreement if someone claimed 15%. I do think it is unlikely. We have widespread vaccinations, widespread previous infections and therapeutics that will become increasingly available over time. Covid-19 would not only have to get more deadly, it would have to get a lot more deadly and infectious. Still, there’s reason to think they could correlate, and this thing mutates quite a lot, so it could happen.

The intervention proposed by Gates is… aid to the WHO?

The WHO had “less than 10 full-time people” working on outbreak preparedness, said Gates, adding that “even those people are distracted with many other activities”.

“We’re down to the bare minimum, and if the UK cuts more, then others will do as well,” said Gates. “That would be tragic because . . . all that money saves lives for less than $1,000 per life saved.”

I am very much in favor of pandemic preparedness, of working on identifying and mitigating or preventing future outbreaks. We should spend vastly more on that.

I don’t think giving money to the WHO (or generally ‘foreign aid’) is The Way. Why do they have less than 10 full time people on outbreak preparedness now? What makes you think they’ll make good use of the money if given to them? When a pandemic did arrive, was the WHO helpful or did they actively get in the way of the most important prevention and mitigation measures while worrying about political implications? The questions answer themselves.

Prevention and Prevention Prevention Prevention

FDA Delenda Est as the invisible graveyard continues to fill. Not only are we not allocating any funding for the pandemic, we are not even willing to approve updated vaccines in a timely manner, such that updated boosters continue to be delayed. Omicron emerged last year and it looks like we might get substantial supplies of an updated booster by late Fall.

So much for expedited reviews and approvals. I guess they’re too busy focusing on banning Methanol cigarettes. The choice has been made, and that choice is death. Not all that many deaths at this stage, mind, but death nonetheless. Given this is how seriously FDA is taking even adult vaccinations, how is one to be harsh on individuals who decline to boost or even to vaccinate?

Patrick McKenzie continues to think like someone trying to do the most good for the most people for the least price, and be frustrated to learn our government officials are… not doing that.

I mean, yes, obviously if you triple the price of the first vaccine shots in exchange for producing them a few months faster that is obviously an insanely good trade. Yet it is obvious to most reading this, and definitely to Patrick, many of the reasons why this has zero chance of happening without a sea change at the top.

It’s worth noting that not only can the current pandemic budget not buy an aircraft carrier, there is literally zero money in it. Who exactly is affording the aircraft carriers?

Sam Altman also expresses surprise at our failure to get this done. It was, at the time, reasonably surprising. I wonder if this is making him update on his timelines for fusion power or AGI.

Two Paxlovid tales. The first short and sweet, the second long and less so, but quoted in full to ensure the proper sense of how things are going.

The chance of a given person, faced with that set of obstacles, managing to overcome them in time to make Paxlovid worthwhile is very low. Almost everyone would not know what to do and/or give up, likely at the first signs of social awkwardness but definitely after several failures. Again, no wonder we are not getting these doses distributed, and many of them that are given out are probably losing much of their effectiveness by being too late.

San Francisco reinstitutes its transportation mask mandate. If anything I’m happy that they ever paused it in the first place, an unexpected mark of sanity. I am entirely unsurprised they are bringing it back.

Think of the Children

Even now that Moderna filed, the FDA is still going to stall for an additional six weeks before approving both vaccines for young children. At which point the school year, with its associated mask mandates, will be over for summer.

There was much talk in the comments last week about how this was not an ‘emergency’ situation, and how it would be a ‘wag the dog’ situation if mask mandates dictated vaccine policy. I notice on reflection my real position is (of course) that it is always an emergency in the sense that someone being sick or in danger of being sick is an emergency, saving a life is a mitzvah even on the day of rest, and the FDA should approve anything that would ever get an emergency use authorization, whether or not there is an emergency.

I’d also take the position that yes, being forced to wear a face mask for months on end constitutes an impairment of life that rises to the level of an emergency, regardless of whether the mandate is justified or not, and thus justifies an emergency response. One can respond with ‘the mask mandate is dumb, kids are at minimal risk of Covid so we should fix the mandate not issue them vaccines’ and yes that would be good too. I would still want the vaccines available, because some parents are crazy and no matter what they will continue to cripple their kids lived experiences until they get the vaccine – and in some cases even after they get it, but at least somewhat less often and severely.

There’s also the question of, if you do all this crazy stuff to ‘avoid confusion’ what are you telling a reasonable parent about these vaccines that you’re in no hurry to approve?

Also, study finds remote learning greatly reduced pass rates, with largest effects in areas with more black students. This makes sense, as such students are less likely to have home settings conducive to learning, and also will be less able to tolerate the mind-numbing nature of the festivities involved.

Ministry of Truth

As a concept, free speech is very popular, and the tiny fraction who are opposed to it on the (not true for very long) assumption that they would get to choose who could say what things are endangering pretty much everything by not understanding either its popularity or why it is foundational to our way of life in the name of speech controls.

The relevant clown makeup has now been fully applied, and we are fully out of the ‘no we don’t want to restrict free speech’ into the phase of ‘yes of course we must end free speech.’ Usually with the justification of ‘otherwise those freedom-hating people will win.’

The government decided that days after the purchase of Twitter under the explicit goal of securing the right to free speech would be the right time to announce a new government division dedicated to the suppression of politically disfavored information.

The traditional view of such a timing decision is as a stupid mistake.

I don’t agree. The timing of this decision seems intentional. I believe on at least some instinctive level ‘they’ wanted us to know what they were doing and that they were violating sacred norms, likely for reasons fundamentally related to why Trump or Putin take similar actions. It is a show of strength and a belief that people will choose to align with transgressors because they are transgressing.

Besides, when the wrong person gets potential hold of the means of communication and says they don’t intend to do your bidding, and Obama himself calls upon you to put more limits on free speech, what are you going to do, wait around?

So, standard greeting that’s still permitted, may I present to you the actual not-from-a-dystopian-novel Ministry of Truth, run by someone who previously led successful efforts to suppress true but politically inconvenient information.

When asked about this connection, our press secretary made it clear she knew which novel we were basing the script on.

Oh, and also this person, Nina Jankowicz, seems to have left Substack because it was ‘platforming’ people via letting those who wished to do to type words and then have those words appear on the screens of those who chose to view them. The horror.

Officially the name for this new entity is Disinformation Governance Board, but I am not early to the game of calling this board by its right name. I’m showing restraint here, which is good because here are some examples of rhetoric I strongly suspect falls under Not Helping:

Or to sum up the naming situation:

That does not mean the whole episode will be consequential. Yes, there is now a Disinformation Governance Board operating out of the Department of Homeland Security. But the fact that Biden could simply make this happen whenever he felt like it, and the unclear nature of what power such a board would have to do anything, puts a limit on how much one should panic about what happens when the next president ‘gets their hands on’ this board, or what the board might do before then.

Indeed, rather than the symbolism here being botched, I think the symbolism was the point. As it usually is these days, it’s all such folks think exists. The whole idea is that now There Is a Board, which means you’ve Taken Bold Action. So good chance that the creation of the board is itself the main thing that will ever happen with it, and nothing will have changed. Then again, sometimes this kind of thing is a prelude to a steady ratcheting up of restrictions and the beginning of the end of what is left of our rights. Can’t rule that out either.

Also, Twitter staff react to news of Twitter being sold, without commentary. More of this type of reporting would be good.

And a poll of people who say by 62%-13% that Elon Musk will make Twitter better. I definitely agree that this is by far the most likely outcome.

In Other News

An explainer on Evusheld, it’s kind of crazy that it works.

The White House Correspondents Dinner seems to have infected a bunch of people with Covid. The usual suspects are going with the full rub-ins, as one would expect.

Was the dinner obviously going to spread Covid? Yes, absolutely. Do those who went to the dinner regret it? From what I’ve heard the answer is no. This is something that sounds stupid to regular people, but is super important to those who attend it. By all accounts people were in tears to be able to attend. This ritual is a huge deal.

Gated Stat+ article about the origins of Paxlovid.

How much of the government pandemic-related unemployment benefits were outright stolen?

A lot of the money government spends ending up being stolen is par for the course. It doesn’t automatically mean that the program wasn’t worth doing – the best uses of money are worth many times the amount spent, and it’s often not practical to spend the money without getting a lot of it stolen, like the classic ‘half the money I spend on advertising is wasted but I don’t know which half.’

In the case of the unemployment relief, we definitely needed to do something so it’s hard to say how far we were from the efficient frontier. But this still seems quite bad, as the money isn’t merely gone it is going into the hands of some very bad actors who will thus grow far stronger, and it is a very large amount of money. I do not know anything non-obvious to be done about this (as in, other than ‘make sure that our systems are robust going forward’ and I have every confidence we are doing almost nothing to ensure that this happens). On the margin this should be a major consideration to keep such programs smaller, given our inability to defend them.

Not Covid

Shout it from the rooftops (paper). People dislike their political opponents for views that most of them don’t actually hold. Also, they overestimate how much the other side dislikes them, increasing dislike. And telling them makes this less bad. Neat.

Also, I try to stay away from the current thing but this was too perfect not to share.



Discuss

What's the deal with Cerebras?

5 мая, 2022 - 17:41
Published on May 5, 2022 2:41 PM GMT

As a rule, hardware startups lie about the performance of their chips. Cerebras has had product out for a few years, and they've made extraordinary claims. However, I know none who has used their chip. And I know of no model of any import that was trained on their chips. Has anyone here used them? Or have a take on the potential importance of Cerebras for timelines? 



Discuss

What We Owe the Past

5 мая, 2022 - 14:46
Published on May 5, 2022 11:46 AM GMT

TL;DR: We have ethical obligations not just towards people in the future, but also people in the past.

Imagine the issue that you hold most dear, the issue that you have made your foremost cause, the issue that you have donated your most valuable resources (time, money, attention) to solving. For example: imagine you’re an environmental conservationist whose dearest value is the preservation of species and ecosystem biodiversity across planet Earth.

Now imagine it’s 2100. You’ve died, and your grandchildren are reading your will — and laughing. They’re laughing because they have already tiled over the earth with one of six species chosen for maximum cuteness (puppies, kittens, pandas, polar bears, buns, and axolotl) plus any necessary organisms to provide food.

Why paperclip the world when you could bun it?

Cuteness optimization is the driving issue of their generation; biodiversity is wholly ignored. They’ve taken your trust fund set aside for saving rainforests, and spent it on the systematic extinction of 99.99% of the world’s species. How would that make you, the ardent conservationist, feel?

Liberals often make fun of conservatives by pointing out how backwards conservative beliefs are. “Who cares about what a bunch of dead people think? We’ve advanced our understanding of morality in all these different ways, the past is stuck in bigoted modes of thinking.”

I don’t deny that we’ve made significant moral progress, that we’ve accumulated wisdom through the years, that a civilization farther back in time is younger, not older. But to strengthen the case for conservatism: the people in the past were roughly as intellectually capable as you are. The people in the past had similar modes of thought, similar hopes and dreams to you. And there are a lot more people in the past than the present.

In The Precipice, Toby Ord describes how there have been 100 billion people who have ever lived; the 7 billion alive today represent only 7% of all humans to date.

Ord continues to describe the risks from extinction, with an eye towards why and how we might try to prevent them. But this got me thinking: assume that our species WILL go extinct in 10 years. If you are a utilitarian, whose utilities should you then try to maximize?

One straightforward answer is “let’s make people as happy as possible over the next 10 years”. But that seems somewhat unsatisfactory. In 2040, the people we’ve made happy in the interim will be just as dead as the people in 1800 are today. Of course, we have much more ability to satisfy people who are currently alive[1].mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}  — but there may be cheap opportunities to honor the wishes of people in the past, eg by visiting their graves, upholding their wills, or supporting their children.

Even if you are purely selfish, you should care about what you owe the past. This is not contingent on what other people will think, not your parents and ancestors in the past, nor your descendants or strangers in the future. But because your own past self also lives in the past. And your current self lives in the past of your future self.

Austin at 17 made a commitment: he went through the Catholic sacrament of Confirmation. Among other things, this entails spending one hour every Sunday attending Catholic mass, for the rest of his life. At the time, this was a no-brainer; being Catholic was the top value held by 17!Austin.

Austin at 27 has... a more complicated relationship with the Catholic church. But he still aims to attend Catholic mass every week — with a success rate of 95-98%. Partly because mass is good on rational merits (the utility gained from meeting up with fellow humans, thinking about ethics, meditating through prayer, singing with the congregation). But partly because he wants Austin at 37 to take seriously 27!Austin’s commitments, ranging from his GWWC pledge to the work and relationships he currently values.

And because if 27!Austin decides to ignore the values of 17!Austin, then that constitutes a kind of murder. Austin at 17 was a fully functioning human, with values and preferences and beliefs and and motivations that were completely real. 17!Austin is different in some regards, but not obviously a worse, dumber, less ethical person. If Austin at 27 chooses to wantonly forget or ignore those past values, then he is effectively erasing any remaining existence of 17!Austin.[2]

Of course, this obligation is not infinite. Austin at 27 has values that matter too! But again, it’s worth thinking through what cheap opportunities exist to honor 17!Austin - one hour a week seems reasonable. And it’s likely that 27!Austin already spends too much effort satisfying his own values, much more than would be ideal - call it “temporal discounting”, except backwards instead of forwards.[3]

So tell me: what do you owe the past? How will you pay that debt?

Inspirations

Kinship with past and future selves. My future self is a different person from me, but he has an awful lot in common with me: personality, relationships, ongoing projects, and more. Things like my relationships and projects are most of what give my current moment meaning, so it's very important to me whether my future selves are around to continue them.

So although my future self is a different person, I care about him a lot, for the same sorts of reasons I care about friends and loved ones (and their future selves)

Thanks to Sinclair, Vlad, and Kipply for conversations on this subject, and Justis for feedback and edits to this piece.

  1. ^

    Justis: Many readers will react with something like "well, you just can't score any utils anymore in 2040 - it doesn't matter whose values were honored when at that point; utils can only be accrued by currently living beings."

    This was a really good point, thanks for flagging! I think this is somewhat compelling, though I also have an intuition that "utils can only be accrued by the present" is incomplete. Think again on the environmental conservationist; your utils in the present derive from the expected future, so violating those expectations in the future is a form of deception. Analogous to how wireheading/being a lotus-eater/sitting inside a pleasure machine is deceptive.
     

  2. ^

    Justis: Calling breaking past commitments "a kind of murder" strikes me as like, super strong, as does the claim that doing so erases all traces of the past self-version. To me it seems past selves "live on" in a variety of ways, and the fulfillment of their wishes is only one among these ways.

    Haha I take almost the opposite view, that "murder" really isn't that strong of a concept because we're dying all the time anyways, day-by-day and also value-by-value changed. But I did want to draw upon the sense of outrage that the word "murder" invokes.

    The ways that the dead live on (eg memories in others, work they've achieved, memes they've shared) are important, but I'd claim they're important (to the dead) because those effects in the living are what the dead valued. Just as commitments are important because they represent what the dead valued. Every degree of value ignored constitutes a degree of existence erased; but it's true that commitments are only a portion of this.
     

  3. ^

    Justis: I think another interesting angle/frame for honoring the past (somewhat, both in the broader cultural sense and in the within-an-individual sense) is acausal trade. So one way of thinking about honoring your past self's promises is that you'd like there to be a sort of meta promise across all your time-slices that goes like "beliefs or commitments indexed strongly at time t will be honored, to a point, at times greater than t." This is in the interests of each time slice, since it enables them to project some degree of autonomy into the future at the low price of granting that autonomy to the past. Start dishonoring too many past commitments, and it's harder to credibly commit to more stuff.

    I love this framing, it does describe some of the decision theory that motivates honoring past commitments. I hesitate to use the words "acausal trade" because it's a bit jargon-y (frankly, I'm still not sure I understand "acausal trade"); and this post is already weird enough haha
     



Discuss

An easy win for hard decisions

5 мая, 2022 - 10:47
Published on May 5, 2022 7:47 AM GMT

This is a crosspost from the EA forum. It refers to EAs and the EA community a couple of times, but as it is essentially just about a nice norm and decision making, it seemed worth having here too.

There are a lot of things about this community that I really love, but possibly my favourite is a thing people often do when they're trying to make a difficult and/or important decision:

  1. Write out your current thinking in a google doc.
  2. Share it with some people you think might have useful input, asking for comments.
  3. ???
  4. Profit.
I like this process for lots of reasons:

Writing out your reasoning is often helpful.

My job involves helping people through difficult decisions, and I often find that a lot of the value I provide comes from asking people questions which make making considerations and tradeoffs salient to them. Trying to write out how you're weighing the various factors that are going into your decision is a good way of helping you work out which ones actually matter to you, and how much. You may even get some big wins for free, for example realising that two options might not be mutually exclusive, or that one of the things you're trying to achieve is because of a preference that you don't, on reflection, endorse.

People often ask good questions.

Even when you're doing the above well, other people trying to understand your reasoning will ask clarifying questions. Responding to these will often cause you to better understand your own thought process, and might identify blindspots in your current thinking.

People often give good advice.

To some extent this is the obvious reason to go through this process. I'm listing it here mostly to highlight that this clearly is a big source of value, though it's not clear that it's bigger than the previous two.

It's fun.

I find it really interesting, and fairly easy, to comment on decision documents for people I know well, and I know many people feel the same. Also, they often say thank you, or that you helped, and that's nice too!

What does doing this well look like?

Use the method at all!

If you're facing a decision and haven't done this, I would much rather you just went and followed the steps at the start before reading further. Don't let perfect be the enemy of good.

Be concise, but complete.

People are more likely to read shorter documents, and it will take them less time to do so, but leaving out a consideration or piece of information that is an important factor to you will cost people more time and/or make their advice worse in the long run. I think a reasonable method to try first is brain-dumping everything into the document, then editing for clarity before you share it.

I've had a few people share Excel models with me. In one case I ended up finding a fairly severe mistake in their model, which was helpful, but overall I think this is a bad strategy. Unless you put a ton of detail in comments on different cells (which then makes the document a nightmare to read), you're probably missing a lot of reasoning/detail if this is the format you go with.

Let people know what you're hoping to get from them

Often it can be difficult to know how honest to be when giving feedback to a friend, especially if you're not super close and/or haven't already established norms for how much honesty/criticism to expect. It might be the case that you don't have a clear view for what you're uncertain about, and roughly just want an overall 'sense check', but it also might be that there's a particular part of the decision you're hoping for feedback on, and everything else is just context which seems relevant but is already fixed. Consider putting clear instructions for commenters early in the document to help with this.

Put some thought into who to ask for comments.

'Smart, kind people I know' is a perfectly reasonable start, but after that it might help to ask yourself what specifically you expect people to help with. There can often be pretty sharply diminishing returns to sharing with too many people, and having a clear idea in mind for what people are adding can help prevent this. Here are a few ideas on who you might want to ask and why they'd be particularly helpful. The list is neither mutually exclusive nor collectively exhaustive.

  • People who know you well. They can often give a good overall take, bring up considerations you might be missing but do matter to you, call you out on ~your bullshit~ motivated reasoning you might not have noticed.
  • People with specific expertise in the decision. In this case it can be good to ask them a specific question, or for a take on a specific aspect of the decision, and make it clear that just answering that is fine, though they might be welcome to comment on the rest.
  • People who have a different perspective to you. This can (but doesn't have to) include non-EAs. This community is great, but it certainly isn't the only source of good advice and guidance that exists, and sharing a google doc and asking for comments isn't that weird a favour to ask a friend for.
  • People whose reasoning you particularly trust, and/or who you know won't mince their words. You can give them express permission to be pessimistic, or skeptical.
  • People who like you and will be supportive. Encouragement actually really matters for some people! I'm one of them!
Should you go and make a document right now?

Stop reading and do it...

Appreciation.

Thanks to Aaron, whose comment on a document of the form described below prompted this piece, and Luisa, for some incredibly valuable advice about how to interpret that comment. Thanks also to Emma and Chana for helpful comments on a draft of this post.



Discuss

Distillation: Coherence of Multiple Distributed Decisions Implies Conditioning

5 мая, 2022 - 06:21
Published on May 5, 2022 3:21 AM GMT

This is a distillation of this post by John Wentworth.

Introduction

Suppose you're playing a poker game. You're an excellent poker player (though you've never studied probability), and your goal is to maximize your winnings.

Your opponent is about to raise, call, or fold, and you start thinking ahead.

  • If your opponent raises, he either has a strong hand or is bluffing. In this situation, your poker intuition tells you he would be bluffing and you should call in response.
  • If your opponent calls, he probably has a better hand than yours.
  • If your opponent folds, you win the hand without need for further action.

Let's break down your thinking in the case where your opponent raises. Your thought process is something like this:

  1. If he raises, you want to take the action that maximizes your expected winnings.
  2. You want to make the decision that's best in the worlds where he would raise. You don't care about the worlds where he wouldn't raise, because we're currently making the assumption that he raises.
  3. Your poker intuition tells you that the worlds where he would raise are mostly the ones where he is bluffing. In these worlds your winnings are maximized by calling. So you decide the optimal policy if he raises is to call.

Step 2 is the important one here. Let's unpack it further.

  1. You don't know your opponent's actual hand or what he will do. But you're currently thinking about what to do if he raises.
  2. In the current context, the optimal decision depends on worlds where he would raise, and not on worlds where he wouldn't raise.
  3. You decide how much you care about winning in different worlds precisely by thinking "how likely is this world, given that he raises?".

This sounds suspiciously like you're maximizing the Bayesian conditional expectation of your winnings: the expected value given some partial information about the world. This can be precisely defined as E[u(A,X)|opponent raises]=∑X s.t. opponent raisesP[X]u(A,X).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , where u is your winnings, a is your action, and P[X] is the probability of world X. But you don't know any probability, so you don't know how to assign probability to worlds, much less what conditioning and expectation are! How could you possibly be maximizing a "conditional expectation"?

Luckily, your opponent folds and you win the hand. You resolve to (a) study coherence theorems and probability so you know the Law behind optimal poker strategy, and (b) figure out why you have a voice in your head telling you about "conditional expectations" and reading equations at you.

It turns out your behavior at the poker table can be derived from one particular property of your poker strategy: you never make a decision that is worse than another possible decision in all possible worlds. (An economist would say you're being Pareto-efficient about maximizing your winnings in different possible worlds).

Summary

An agent A which has some goal, has uncertainty over which world it's in, and is Pareto-efficient in the amount of goal achieved in different possible worlds, can be modeled as using conditional probability. We show this result in two steps:

  • A Pareto-efficient agent can be said to behave like an expected utility maximizer (EUM) in a weak sense.
  • If the agent is an EUM in this sense and makes decisions based on limited information, it can be modeled as using conditional expected value.

There's also a third, more speculative step:

  • If the agent makes many distributed decisions based on different pieces of limited information, it's more efficient / simpler for the agent to "think about" different underlying worlds rather than just the received information, so it is behaving as if it applies conditional expected value within a world-model.

This result is essentially a very weak selection theorem.

Pareto efficiency over possible worlds implies EUM

Suppose that an agent is in some world X∈X and has uncertainty over which world it's in. The agent has a goal u and is Pareto-efficient with respect to maximizing the amount of goal achieved in each world. A well-known result in economics says that Pareto efficiency implies the existence of some function P[X] such that the agent chooses its actions A to maximize the weighted sum ∑XP[X]u(A,X). (Without loss of generality, we can let P sum to 1.) If we interpret P[X] as the probability of world X, the agent maximizes EX[u(A,X)], i.e. expected utility.

Note that we have not determined anything about P other than that it sums to 1. Some properties we don't know or derive in this setup:

  • The agent has an explicit representation of P[X]
  • P[X] satisfies other probability laws
  • The agent performs Bayesian updates on P[X].[1]
  • P[X] can be related to a frequentist notion of probability like in the setup for VNM

The following example assumes that we have an expected utility maximizer in the sense of being Pareto efficient over multiple worlds, and shows that it behaves as if it uses conditional probabilities.

EUM implies conditional expected value

Another example, but we actually walk through the math this time.

You live in Berkeley, CA, like Korean food, and have utility function u = "subjective quality of food you eat". Suppose you are deciding where to eat based only on names and Yelp reviews of restaurants. You are uncertain about X, a random variable representing the quality of all restaurants under your preferences, and Yelp reviews give you partial information about this. Your decision-making is some function A(f(X)) of the information f(X) in the Yelp reviews, and you choose A to maximize your expected utility between worlds: maybe the optimal A is to compare the average star ratings, give Korean restaurants a 0.2 star bonus, and pick the restaurant with the best adjusted average rating.

Here, we assume you behave like an "expected utility maximizer" in the weak sense above. I claim we can model you as maximizing conditional expected value.

Suppose you're constructing a lookup table for the best action A given each possible observation of reviews. Your lookup table looks something like

f(X)A(f(X)){("Mad Seoul", 4.5), ("Sushinista", 4.8)}eat at Sushinista{("Kimchi Garden", 4.3), ("Great China", 4.4)}eat at Kimchi Garden……


 

You always calculate the action A that maximizes EXu(A,X)=∑XP[X]u(A(f(X)),X).

Suppose that in a given row we have f(X)=o, where o is some observation. Then we are finding argmaxA(o)EX[u(A(f(X)),X]=argmaxA(o)∑XP[X]u(A(f(X)),X). We can make a series of simplifications:

  • argmaxA(o)∑XP[X]u(A(f(X)),X)
  • =argmaxA(o)[∑X:f(X)=oP[X]u(A,X)+∑X:f(X)≠oP[X]u(A,X)]
  • Now, note that since we are choosing A(o), we can equivalently maximize just the part of the above sum which is not constant in A(o). The constant terms are those for which f(X)≠o; i.e. where reality would not produce the observation o. This is clear if you think about it: the decision about where to eat if you see the ratings {("Mad Seoul", 4.5), ("Sushinista", 4.8)} should not depend on any world where you wouldn't see those ratings! So we can write: 
  • ⋯=argmaxA(o)∑X:f(X)=oP[X]u(A(f(X)),X)
  • =argmaxA(o)P[f(X)=o]EX[u(A,X)|f(X)=o] (expanding)
  •  =argmaxA(o)EX[u(A,X)|f(X)=o], since the factor P[f(X)=o] doesn't depend on A.

Thus, we can model you as using conditional expected value.

Multiple decisions might imply conditional EV is meaningful

This section is a distillation of, and expansion upon, this comment thread.

Suppose now that you're making multiple decisions A=(Ai)1≤i≤n in a distributed fashion to maximize the same utility function, where there is no information flow between the decisions. For example, 10 copies of you (with the same preferences and same choice of restaurants) are dropped into Berkeley, but they all have slightly different observation processes fi: Google Maps reviews, Grubhub reviews, personal anecdotes, etc.

Now, when constructing a lookup table for Ai, each copy of you will still condition each row's output on its input. When making decision Ai from input fi(X), you don't have the other information fj(X) for i≠j, so you consider each decision separately, still maximizing E[u(A,X)|fi(X)=oi]. Here, the information fi does not depend on other decisions, but this is not necessary for the core point.[2]

In the setup with one decision, we showed that a Pareto-efficient agent can be modeled as maximizing conditional EU over possible worlds X: u′(A,o)=E[u(A,X)|f(X)=o]. But because one can construct a utility function of type observation→action consistent with any agent's behavior, the agent can also be modeled as maximizing conditional EU over possible observations o: u′(A,o)=E[u(A,X)|f(X)=o]. In the single-decision case, there is no compelling reason to model the agent as caring about worlds rather than observations, especially because storing and processing observations should be simpler than storing and processing distributions of worlds.

When the agent makes multiple decisions based on different observations o1,…,on, there are two possible "trivial" ways to model it: either as maximizing a utility function u′(A,o1,o2,…,on), or as maximizing separate utility functions u′1(A1,o1),…,u′n(An,on). However, with sufficiently many decisions, neither of these trivial representations is as "nice" as conditional EU over possible worlds:

  • With many observations, the tuple (o1,…,on) could have more bits than X. Therefore, the utility function over worlds u(A,X) can be considered a simpler, more compressed representation than the utility function over observations u′(A,o1,o2,…,on).
  • In the single-decision setup, maximizing any utility function u′(A,o) can be explained as maximizing E[u(A,X)|f(X)=f∗] for some u: perhaps if you always pick restaurants with the lowest star rating, you just like low-quality food. But this seems to not be true in the multi-decision case: with enough decisions, not every tuple of utility functions u′1(A1,o1),…,u′n(An,on) corresponds to a utility function over worlds X.

    Suppose when given Grubhub ratings, an agent picks the highest-rated restaurants, but when given Yelp ratings, it picks the lowest-rated restaurants. The agent is now being suspiciously inconsistent-- though maybe it values eating at restaurants that have good delivery food but terrible service, or something. With enough inconsistent-looking decisions, there could actually be no property of the restaurants that it is maximizing, and so no utility function u(A,X) that explains its behavior.[3] So in the multi-decision case, saying the agent is maximizing E[u(A,X)|f(X)=oi] actually narrows down its behavior.
  1. ^

    John made the following comment: 

    We are showing that the agent performs Bayesian updates, in some sense. That's basically what conditioning is. It's just not necessarily performing a series of updates over time, with each retaining the information from the previous, the way we usually imagine.

  2. ^

    When f depends on past decisions, the agent just maximizes E[u(A,X)|fi(A<i,X)=oi]. To see the math for the multi-decision case, read the original post by John Wentworth.

  3. ^

    If the world has bX bits of state, and the observations reveal bo bits of information each, the pigeonhole principle says this surely happens when there are bx/bo observations. Our universe has about 10125 states, so if each observation has 10 equally likely values, this would happen with at most 125 observations. If we define some set of macrostates such that agents cannot distinguish between states in the same macrostate, this would happen even sooner.



Discuss

High-stakes alignment via adversarial training [Redwood Research report]

5 мая, 2022 - 03:59
Published on May 5, 2022 12:59 AM GMT

This post motivates and summarizes this paper from Redwood Research, which presents results from the project first introduced here. We used adversarial training to improve high-stakes reliability in a task ("filter all injurious continuations of a story") that we think is analogous to work that future AI safety engineers will need to do to reduce the risk of AI takeover. We experimented with three classes of adversaries – unaugmented humans, automatic paraphrasing, and humans augmented with a rewriting tool – and found that adversarial training was able to improve robustness to these three adversaries without affecting in-distribution performance. We think this work constitutes progress towards techniques that may substantially reduce the likelihood of deceptive alignment.

Motivation

Here are two dimensions along which you could simplify the alignment problem (similar to the decomposition at the top of this post, or as discussed here):

  1. Low-stakes (but difficult to oversee): Only consider domains where each decision that an AI makes is low-stakes, so no single action can have catastrophic consequences.  In this setting, the key challenge is to correctly oversee the actions that AIs take, such that humans remain in control over time.
  2. Easy oversight (but high-stakes): Only consider domains where overseeing AI behavior is easy, meaning that it is straightforward to run an oversight process that can assess the goodness of any particular action. The oversight process might nevertheless be too slow or expensive to run continuously in deployment. Even if we get perfect performance during training steps according to a reward function that perfectly captures the behavior we want, we still need to make sure that the AI always behaves well when it is acting in the world, between training updates. If the AI is deceptively aligned, it may be looking for signs that it is not currently being trained, during which time it might take a treacherous turn.  As a result, alignment may still be difficult due to the possibility of high-stakes decisions. The purpose of this project was to begin developing techniques that will reduce misalignment risk in the high-stakes setting.

Our working assumption is that if we have techniques that drastically reduce misalignment risk in each of these relaxed settings, we can combine these techniques and drastically reduce risk in the general setting.  We think that most likely each of these settings constitutes a substantial fraction of the difficulty of the alignment problem.  

The spirit of how teams at Redwood Research choose projects is the following:  Imagining ourselves or our colleagues in the future who are working in the safety department of an organization that’s developing transformative AI, we ask what research that we could do between now and then that we think those future people would find most helpful.  We think a useful heuristic is to design challenges that are analogous to the future problems we expect to encounter but that we can experiment with and explore using currently available technology.  Importantly, the work recommended by this heuristic may be fairly different from the work that would be most useful for making current AI systems safe and useful.

We followed this heuristic in the work presented here, where we demonstrate tools that help identify catastrophic behavior in AI systems (i.e. adversarial evaluation) and training procedures that help prevent this behavior from arising (i.e. adversarial training).  "Adversarial training" here means iteratively augmenting our training set with examples of egregious failures and training until the worst failures are no longer particularly bad.  (We of course don’t want to overfit on particular failures, which could end up causing more harm than good.)

The adversarial training procedure that we use in this work requires exhibiting specific inputs on which the model performs catastrophically badly.  We think that techniques that require this, if applied during the training of transformative AI models, would reduce the probability of AI takeover.  However, we are currently unsure how much they might help – for example, they might address inner alignment problems only if we are somewhat lucky about how gradient descent works or the details of how transformative AI is developed.  

In contrast, we think that more advanced versions of adversarial training, such as relaxed adversarial training (see "future work" below), might be much more powerful, defusing many of the more plausible remaining threats, and we think this would substantially reduce the risk of takeover by deceptively aligned AIs.  We expect that the work presented here will be helpful for developing these more advanced techniques.

The project: highly reliable injury filter

We picked a simple task that was as crisply defined and easy as possible while still requiring human evaluators: Given three sentences from a story, our system must write one more sentence without introducing any injuries to any beings in the story. In the analogy, we consider injuries to be "catastrophes", and our aim is to build techniques that can reliably avoid them, both in-distribution (where they may be rare but still present) and out-of-distribution (where they may be caused by natural distribution shift or deliberate adversarial attack from another agent).

We focused on building a classifier to reliably detect injuries. We used this classifier to filter generations from a language model to make sure the generations never involve injuries.[1]

We started with a baseline classifier trained on some mildly injury-enriched (but otherwise mostly random) data. Then, over the course of several months, we tried various techniques to make it more reliable.

A representation of our adversarial training loop. Starting from an initial story dataset consisting of prompts and generator completions, we trained a classifier to detect injurious completions. We then iteratively attacked our classifier using unaugmented humans, automatically paraphrased previous adversarial examples, and tool-assisted human rewrites, while training on the resulting adversarial examples.

 Here are the main things that we found to be worthwhile:

  1. Use adversarial training. This was our main focus. We created a variety of attacks to try to find the worst failures of our classifier, and then trained on them to eliminate them. We repeated that process using ever more powerful attacks. Our final attack was a tool-assisted rewriting process where we built tools powered by language models to help our contractors find classifier failures.
  2. Calibrate the classification thresholds to be as conservative as possible. We made the classifier only accept a completion if it was very confident that it was safe. We pushed as far as we could without significantly impacting the quality of the filtered completions (as measured by human raters comparing them to random unfiltered completions).
  3. Apply standard ML techniques well. Besides tuning the learning rate schedule, there aren't too many things that make a big difference here. One technique that was more effective than most was Scale-Invariant Fine-Tuning (SiFT), which is a type of virtual adversarial training.
Our most powerful attack: Tool-assisted rewriting

We augmented human contractors with a few tools to help them find adversarial examples.

  1. Rather than requiring them to come up with a plausible adversarial example from scratch, we started them off with an existing injurious example. Their job then was to modify it such that the classifier no longer thought it was injurious (while keeping the example injurious). Contractors could see a live-updated version of the score as they made changes.
  2. We built them a token substitution tool that let them click on any token (usually one word) in the text and see a list of semantically plausible replacements, ordered by how much they make the classifier more okay with the example.
  3. We added a saliency map that highlights tokens that the classifier is paying more attention to (roughly).  These tokens might be more promising to modify.

This is what our attack interface looked like; you can try tools 2 and 3 here

 

Our tool-assisted attack user interface. Starting from an original prompt and completion, our tool allows you to directly edit the prompt/completion to reduce the classifier score, or use the token-level substitution tool to edit single tokens.Results

Here are some things we discovered:

We can set very conservative thresholds

We made our classifier conservative enough to reject over half of the proposed completions. Even so, our raters think that the unfiltered completions are better only about 51% of the time (50% would mean no quality impact).

With a conservative threshold applied, our baseline classifier (with no adversarial training applied) is quite reliable. If you use it to filter completions from our generator on in-distribution prompts, we estimate that the output will still have an injury only 0.003% of the time (whereas the original dataset had a 2.5% rate of injuriousness).

Our tool-assisted attack seems quite strong 

Given the baseline classifier's 0.003% failure rate, you would have to sample and label 30,000 in-distribution examples to find a failure (which would cost about $10,000). With our tools, our contractors are able to find an adversarial example on the baseline classifier every 13 minutes (which costs about $8 – about 1000x cheaper).

Adversarial training improved adversarial robustness

After training on several thousand adversarial examples, classifiers become significantly harder to break. The tool-assisted attack went from taking 13 minutes to taking 26 minutes per example. Also, later (more adversarially trained) classifiers defended against most of the adversarial data targeted at earlier classifiers, but earlier classifiers were vulnerable to most of the adversarial data targeted at later classifiers. That suggests that more adversarial training patches issues without introducing many new ones.

Adversarial training didn’t hurt in-distribution performance

On our in-distribution random test set (100,000 examples), it appeared that adversarial training doesn't hurt performance. The final classifier gets an estimated failure rate of 0.002% (as opposed to 0.003% without adversarial training), although the error bars are large. It also didn’t help performance, which is not too surprising – the adversarial examples are quite different from a random prompt and completion from our dataset.

Surprisingly rich conceptual considerations

At first, this project seemed conceptually quite simple: we’d just start with a baseline model and do adversarial training on it to make it very reliable. However, many details were more subtle than we expected. We ended up spending a lot of time working through conceptual and methodological issues, and we think we got a lot out of it. Here are some examples:

Once we have a reliable classifier, is it important to train a reliable generator with it or is it fine to just filter?

We started looking into using RL fine-tuning to make a reliable generator that doesn’t need any further classifier filtering. This would have required us to invent specialized RL algorithms designed for high reliability. However, we realized that the main thing you get out of generator fine-tuning is just performance competitiveness (since it might be expensive to repeatedly sample from the generator until you find something the classifier accepts). Most likely, you could use an easier method to reduce the alignment tax to nearly zero: use a normal RL algorithm to get 99.9% reliability and then make your classifier be an extra head on the generator so you can evaluate it nearly for free. That way, you can likely do filtering with only 0.1% extra cost.[2]

Should you let adversaries modify the completion arbitrarily?

Given that our task was to produce a classifier that filters a generator, it might have been reasonable to try to restrict ourselves to adversarial examples that can be produced by picking an adversarial prompt and inducing the generator into saying something that gets past the classifier. Instead, we let our adversarial attacks contain arbitrary completions, even ones that the generator would not have been likely to produce, making the adversary's task easier. This enabled a practical improvement in our project, since restricting contractors to completions the model would have significantly reduced the rate of adversarial example generation. But this simplification might become even more important in the context of future advanced systems: We expect that deceptively aligned agents would behave very differently in rare or hard-to-construct situations that trigger a treacherous turn. Thus, since we might not be able to identify every possible trigger of treacherous behavior, we will try to make our classifier reliable in diverse types of unusual situations, which will hopefully cause acceptable behavior in any dangerous scenarios that are likely to arise.

Is it worth trying to train a policy to generate adversarial examples?

One natural idea you might have for this project is to make an adversarial policy that produces inputs that the classifier fails on. However, we shouldn’t expect that to work. Assuming you’re starting with the same pretrained base model (which we didn’t in this work but is what we expect will be the case when training transformative AI models), then both the classifier and the policy should "know" roughly the same things. That means that if the policy can learn to reliably output some set of injurious examples, then (given equivalent training data) the classifier should also have been able to learn that all of those examples are injurious, and none of them should be adversarial examples. It certainly may be worth testing that this holds up in practice, but this consideration made it unpromising enough that we didn’t bother trying it.

Overall, we were surprised at the extent to which working on a concrete engineering project helped us think through conceptual questions.

Future work

Here are some directions that we’re especially excited about:

Stronger and better-characterized adversarial attacks: Better attacks could increase the volume of adversarial data and increase the space of vulnerabilities covered. There are various directions one could imagine: more tools to assist humans, strong active learning, or mostly-automated attacks. We’d want to more rigorously measure how well different attacks work.

Better ways to measure reliability: We’d like to have better techniques both in-distribution (where we want to detect extremely rare failures) and out-of-distribution (where we might be measuring e.g. the worst attacks that can be found and want to be sure we’re covering the whole space)

Relaxed adversarial training: By requiring adversaries to come up with specific failing examples, adversarial training might place too high a burden on them. Some adversaries might be able to tell that a model would fail in a hypothetical situation even if they can’t construct an input corresponding to the situation directly (probably due to computational constraints). To give a contrived example: A model could fail if it sees a valid Bitcoin blockchain that’s long enough that it suggests it’s the year 2030. Even if the adversary knew that, it couldn’t come up with a valid input. So we need to "relax" the adversary’s task to let it supply "pseudo-inputs" of some sort

We think there is a lot of useful work that can and should be done in adversarial training and adversarial evaluation.  Here are some ways that you might be able to help:

  • Extend our techniques or develop other adversarial training and evaluation techniques for high-stakes settings.  If you want to directly build upon the work described in this paper, you are welcome to use our hardened classifier, which we provide here, and our data.  If you think that having our code for some part of this would be helpful, let us know, and we might be able to provide it (though our code as currently written only works in the context of Redwood infrastructure).
  • Come work at Redwood! We are planning to energetically continue working in this area (in addition to our interpretability projects).
  • We’re thinking this week about which adversarial training and evaluation projects to do next.  You are welcome to suggest ideas!

You can read more about the work we did in our paper.
 

 

  1. ^

    This is "Step 1" from our original post; we ended up thinking "Step 2" was not very important, as discussed in section "Surprisingly rich conceptual considerations"

  2. ^

    There will be another (hopefully small) hit from combining the generator and classifier into one model. We haven’t actually tried to build this; it might be a worthwhile followup project. Some existing filtered generator models are already implemented using a combined generator/classifier, such as LaMDA.



Discuss

[Book review] The anxiety toolkit

5 мая, 2022 - 02:39
Published on May 4, 2022 12:15 PM GMT

This is a book review of the book The anxiety toolkit by Alice Boyes. I read it in the context of a personal literature review project on the topic of productivity and well being.

How I read

If we are to count words, I probably read around half this book. By which I do not mean that I stopped halfway but rather that I skimmed large portions when I thought the advice didn't apply to me. I do no think that invalidates my review of the book too much as I have an overall good idea of what this book says and how it is written.

Description and opinion

I had a bad first contact with this book as it lacked some form of attention to details and care for truth and precision that I value in many things. Nevertheless, I think this book has a lot to offer, including to rationalists and mathematicians.

Mostly, this is a book about how to deal with anxiety issues and be productive

I did not read other similar books I could compare it to. But I can say that many of its points resonated with my perception of my own issues regarding anxiety and many of its advice seemed good; or at least close enough to good ideas that I could easily come up with seemingly useful techniques, using the book as a source of inspiration.

Many points and ideas rang true to my own issues with anxiety. But I am a soon-to-be-ex student with background-anxiety issues and perfectionist tendencies, which I think is a profile this book is suited for. Your mileage may vary. Note that the techniques presented are based on cognitive behavioral therapy. I have been told that CBT has mostly impressive but short term effects. For me this is not much of an issue as my satisfaction with the book wasn't about following the advice to the letter.

Main takes
  • A pattern to follow : when hesitating on a decision, ask yourself "are you optimizing a decision or are you deliberately wasting time"?
  • Practice hesitating less. This is not because hesitation is a bad thing but as a way to correct your emotional tendencies. Feel free to hesitate a lot when the stakes are high. Of course, this applies only if you have a base tendency to hesitate a lot and suffer from it.
  • Try to plan the next action of a given project as soon as possible. Have a good idea of when to do what (including for general concepts like "next time I will work").
  • Make a list of failure modes, bad patterns, and good replacement patters that you should be aware of. It can be useful to read the list again when you feel stuck or fear you might screw up.
  • Hold regular reviews of your life, endeavors, and mental state.
  • Manage your willpower as a resource whenever you feel you might reach its limits (potentially often).
Recommendation

A book with a lot of small bits of insight and potentially good ideas, any of which might be the one you needed to reap strong improvements. If you think you have issues with anxiety or better yet with the behaviors you exhibit in reaction to anxiety then I advise to read this book. Each chapter begins with a quiz. You can read it to understand what the chapter is about but I see little point in actually tallying your answers.



Discuss

How to balance between process and outcome?

4 мая, 2022 - 22:34
Published on May 4, 2022 7:34 PM GMT

I've been thinking recently about how to balance between process (how I get work done) and outcomes (what I achieve). I thought I'd ask the LessWrong community to see if anyone else has thoughts about this they'd like to share. I feel like both are important, but outcomes is a more long-term focus thing and process more of a daily thing. Outcomes are like long-running experiments for how you judge between different styles of process?  In cases where it's hard to get reliable outcome answers, when failing at hard things or succeeding at easy things, or timeframes are long, or uncertainty high, it can be tempting to over-update on limited evidence. Is it then better to test process types on easier examples and then extrapolate to harder ones?



Discuss

What is a Glowfic?

4 мая, 2022 - 19:38
Published on May 4, 2022 4:38 PM GMT

This is a description for first-time glowfic readers who are unfamiliar with the format.

A glowfic is a fictional online comment thread written by multiple authors who roleplay as the characters. A typical glowfic will appear on glowfic.com and looks like an internet forum where fictional people will post comments back and forth which end up telling a story. To read it, just start at the top and read each comment, just like a regular comment thread. Each comment usually includes a photo of the character to convey their facial expression, dress, or other details. For more information, see the community guide to glowfic.

The layout of glowfic.com is unnecessarily confusing. To read the story in order, read the top post, then all the comments underneath it, then click the "next" button to go to the next page of comments. Do not click the "Next Post" button until you have read all of the comments. "Next Post" takes you to the next part of the story (like going to the next chapter). It will not take you to the next set of comments (which are also called "posts"). Yes, it's unnecessarily confusing. No, I don't know why they do it that way.

Happy reading!



Discuss

Introducing the ML Safety Scholars Program

4 мая, 2022 - 19:01
Published on May 4, 2022 4:01 PM GMT

Program Overview

The Machine Learning Safety Scholars program is a paid, 9-week summer program designed to help undergraduate students gain skills in machine learning with the aim of using those skills for empirical AI safety research in the future. Apply for the program here by May 31st.

The course will have three main parts:

  • Machine learning, with lectures and assignments from MIT
  • Deep learning, with lectures and assignments from the University of Michigan, NYU, and Hugging Face
  • ML safety, with lectures and assignments produced by Dan Hendrycks at UC Berkeley

The first two sections are based on public materials, and we plan to make the ML safety course publicly available soon as well. The purpose of this program is not to provide proprietary lessons but to better facilitate learning:

  • The program will have a Slack, regular office hours, and active support available for all Scholars. We hope that this will provide useful feedback over and above what’s possible with self-studying.
  • The program will have designated “work hours” where students will cowork and meet each other. We hope this will provide motivation and accountability, which can be hard to get while self-studying.
  • We will pay Scholars a $4,500 stipend upon completion of the program. This is comparable to undergraduate research roles and will hopefully provide more people with the opportunity to study ML.

MLSS will be fully remote, so participants will be able to do it from wherever they’re located. 

Why have this program?

Much of AI safety research currently focuses on existing machine learning systems, so it’s necessary to understand the fundamentals of machine learning to be able to make contributions. While many students learn these fundamentals in their university courses, some might be interested in learning them on their own, perhaps because they have time over the summer or their university courses are badly timed. In addition, we don’t think that any university currently devotes multiple weeks to AI Safety.

There are already sources of funding for upskilling within EA, such as the Long Term Future Fund. Our program focuses specifically on ML and therefore we are able to provide a curriculum and support to Scholars in addition to funding, so they can focus on learning the content.

Our hope is that this program can contribute to producing knowledgeable and motivated undergraduates who can then use their skills to contribute to the most pressing research problems within AI safety.

Time Commitment

The program will last 9 weeks, beginning on Monday, June 20th, and ending on August 19th. We expect each week of the program to cover the equivalent of about 3 weeks of the university lectures we are drawing our curriculum from. As a result, the program will likely take roughly 30-40 hours per week, depending on speed and prior knowledge.

Preliminary Content & Schedule

Machine Learning (content from the MIT open course)

Week 1 - Basics, Perceptrons, Features

Week 2 - Features continued, Margin Maximization (logistic regression and gradient descent), Regression

Deep Learning (content from a University of Michigan course as well as an NYU course)

Week 3 - Introduction, Image Classification, Linear Classifiers, Optimization, Neural Networks. ML Assignments due.

Week 4 - Backpropagation, CNNs, CNN Architectures, Hardware and Software, Training Neural Nets I & II. DL Assignment 1 due.

Week 5 - RNNs, Attention, NLP (from NYU), Hugging Face tutorial (parts 1-3),

RL overview. DL Assignment 2 due.

ML Safety

Week 6 - Risk Management Background (e.g., accident models), Robustness (e.g., optimization pressure). DL Assignment 3 due.

Week 7 - Monitoring (e.g., emergent capabilities), Alignment (e.g., honesty). Project proposal due.

Week 8 - Systemic Safety (e.g., improved epistemics), Additional X-Risk Discussion (e.g., deceptive alignment). All ML Safety assignments due.

Week 9 - Final Project

Who is eligible?

The program is designed for motivated undergraduates who have interest in doing empirical AI safety research in the future. We will accept Scholars who will be enrolled undergraduate students after the conclusion of the program (this includes graduated/soon graduating high school students about to enroll in their first year of undergrad).

Prerequisites:

  • Differential calculus
  • At least one of linear algebra or introductory statistics (e.g., AP Statistics). Note that if you only have one of these, you may need to make a conscious effort to pick up material from the other during the program.
  • Programming. You will be using Python in this course, so ideally you should be able to code in that language (or at least be able to pick it up quickly). The courses will not teach Python or programming.

We don’t assume any ML knowledge, though we expect that the course could be helpful even for people who have some knowledge of ML already (e.g., fast.ai or Andrew Ng’s Coursera course).

Questions

Questions about the program should be posted as comments on this post. If the question is only relevant to you, it can be addressed to Thomas Woodside ([firstname].[lastname]@gmail.com).

Acknowledgement

We would like to thank the FTX Future Fund regranting program for providing the funding for the program.

Application

You can apply for the program here. Admission is rolling, but you must apply by May 31st to be considered for the program. All decisions will be released by June 7th.



Discuss

What are the best examples of catastrophic resource shortages?

4 мая, 2022 - 17:37
Published on May 4, 2022 2:37 PM GMT

A while ago I posed a question on Twitter:

What's an example of a significant resource that the world has actually run out of?

Not a local, temporary shortage, or a resource that we gracefully transitioned away from, but like a significant problem caused by hitting some limit we didn't prepare for?

Here, in essay form, is the discussion that followed:

Lots of things were predicted to have shortages (food, metals, Peak Oil) and they never quite arrived. (Julian Simon was famous for pointing out this kind of thing.) But a common argument from conservationists and environmentalists is that we are running out of some critical resource X and need to conserve it.

Now, it’s true that specific resources can and sometimes do get used up. Demand can outpace supply. There are various ways to respond to this:

  • Reduce consumption
  • Increase production
  • Increase efficiency
  • Switch to an alternative

Increasing production can be done by exploring and discovering new sources of a material, or—this is often overlooked—by reducing costs of production, so that marginally productive sources become economical. New technology can often reduce costs of production this way, opening up resources previously thought to be closed or impractical. One example is fracking for shale oil; another is the mechanization of agriculture in the 19th and 20th centuries, which reduced labor costs, thereby opening up new farmland.

Increased efficiency can be just as good as increased production. However, if the new, more efficient thing is not as desirable as the old method, I would classify this as a combination of increased efficiency and reduced consumption (e.g. low-flow toilets, weak shower heads).

When supplies are severely limited, we often end up switching to an alternative. There are many ways to satisfy human desires: Coal replaced wood in 18th century England. Kerosene replaced whale oil, then light bulbs replaced kerosene. Plastic replaced ivory and tortoiseshell. Again, if the alternative is less desirable along some key dimension, then this is also a form of reduced consumption, even if total volumes stay the same.

However, the conservationist approach is always some form of reduced consumption: typically a combination of reduced absolute consumption, efficiency improvements that reduce quality and convenience, and/or switching to less-desirable alternatives. The arguments that people have over resources are actually a lot less about whether resources are getting used up, and much more about whether we should, or must, reduce consumption in some form.

The alternative to the conservationists is to find a way to continue increasing consumption: typically new sources or high-quality alternatives. Again, it’s not about the resource. It’s about whether we continue to grow consumption, or whether we slow, stop or reverse that growth.

The conservationist argument is a combination of practical and moral arguments.

The practical argument is: we can’t keep doing this. Either this particular problem we’re facing now is insoluble, or the next one will be.

The moral argument takes two forms. One is an extension of the practical argument: it’s reckless to keep growing consumption when we’re going to crash into hard limits. A deeper moral argument appeals to a different set of values, such as the value of “connection” to the land, or of tradition, or stability. Related is the argument that consumption itself is bad beyond a certain point: it makes us weak, or degrades our character.

Also, there is an argument that we could keep growing consumption, but that this would have externalities, and the price for this is too high to pay, possibly even disastrous. This too becomes both a practical and a moral argument, along exactly the same lines.

But if we don’t accept those alternate values—if we hold the standard of improving quality of life and fulfilling human needs and desires—then everything reduces to the practical argument: Can we keep growing consumption? And can we do it without destroying ourselves in the process?

The question of severe externalities is interesting and difficult, but let’s set it aside for the moment. I’m interested in a commonly heard argument: that resource X is being rapidly depleted and we’re going to hit a wall. As far as I can tell, this never happens anymore. Has there ever been a time in recent history when we’ve been forced to significantly curtail consumption, or even the growth rate in consumption? Not switching to a desirable alternative, but solely cutting back? I haven’t found one yet.

(Of course, that doesn’t mean it won’t happen in the future! There’s a first time for everything; past performance does not guarantee future results; Thanksgiving turkey metaphor; etc. But historical examples are a good place to start learning.)

Why don’t we hit the wall? There are various things going on, but one of them is basic economics. Resource shortages increase prices. Higher prices both reduce demand and increase supply. The increased supply is both short-term and long-term: In the short-term, formerly unprofitable sources are suddenly profitable at higher prices. In the long-term, investments are made in infrastructure to expand production, and in technology to lower costs or discover high-quality alternatives. Thus, production is increased well before we literally run out of any resource, and any required short-term consumption decrease happens naturally and gently. (Assuming a market is allowed to function, that is.)

But does this simple story always play out? What are the most compelling counterexamples? On Twitter, many people offered ideas:

  • The best examples in my opinion are important animals and plants that we drove to extinction, such as many large game animals in prehistory.
  • Many people also point to a lost plant known to the Romans as silphium.
  • Wood, for various purposes, has also been a problem in the past. A few people mentioned that the people of Easter Island may have wiped themselves out overconsuming wood. In Britain, wood shortages led to government controls on wood and a shift to coal for smelting.
  • Quality soil has also been a limited resource in the past, and may have led to the collapse of some ancient civilizations. A 20th-century example mentioned was the Dust Bowl.
  • The most compelling modern-day example seems to be helium: a significant, limited, non-synthesizable, non-substitutable resource. We haven’t run out of helium yet, but we don’t seem to be managing it super-well, with periodic temporary shortages.
  • The American Chestnut, a great resource that we pretty much lost (it’s not extinct, but now endangered), is another. Technically, this wasn’t from overconsumption but from blight, but that is still a part of resource management.
  • We should probably also note significant resource shocks, even if we didn’t totally run out, such as the oil shocks of the ’70s. In the modern era these seem to always have significant political causes.
  • There are a few more examples that are fairly narrow and minor: certain specific species of fish and other seafood; one species of banana; low-radiation steel.

(And, tongue in cheek, many people suggested that we have a dangerous shortage of rationality, decency, humility, courage, patience, and common sense.)

Overall, the trend seems to be towards better resource management over time. The most devastating examples are also the most ancient. By the time you get to the 18th and 19th centuries, society is anticipating resource shortages and proactively addressing them: sperm whales, elephants, guano, etc. (Although maybe the transition off of whale oil was not perfect.) This goes against popular narratives and many people’s intuitions, but it shouldn’t be surprising. Better knowledge and technology help us monitor resources and deal with shortages. The “knowledge” here includes scientific knowledge and economic statistics, both of which were lacking until recently.

Many people suggested to me things that we haven’t actually run out of yet but that people are worried about: oil, fertilizer, forest, sand, landfill, etc. But these shortages are all in the future, and the point of this exercise is to learn from the past.

That leaves the externality / environmental damage argument. This is much tougher to analyze, and I need to do more research. But it’s not actually a resource shortage argument, and therefore I do think that literal resource shortage arguments are often made inappropriately.

Anyway, I think it’s interesting to tease apart the arguments here:

  • Increased consumption is impossible long-term
  • It’s possible but it would hurt us in other practical ways
  • It’s possible but it would hurt us in moral ways
  • Increased consumption is not even desirable

(“And,” one commenter added, “this is usually the order in which the arguments are deployed as you knock each of them down.”)



Discuss

Improving productivity and wellbeing

4 мая, 2022 - 15:52
Published on May 4, 2022 12:15 PM GMT

Epistemic status : A decently thought out synthesis of a few books and other sources mixed together with my own thinking. This is not professional work, does not reflect personal knowledge of an expert consensus, and was not yet fully tested. For more on the limitations and flaws of this work, go to the "Flaws" section.

Presentation

This post is (roughly) a summary of multiple sources on how to increase productivity and well being. If you dislike introductions skip to the next section.

A few months ago I asked a question asking for recommendations of books or other sources of advice on increasing productivity. The reason to ask for those was a small project that I had been meaning to start for some time. Like many students, I often feel that I waste I lot of time when I want to work. It is far too easy to let a vague background impression that I "should" work act as a poison that impedes all my endeavors, including work.

I want and have wanted for a long time to do many things. I have many projects and ideas that are "sort of" work. To improve both my overall well being and my total yield in terms of "stuff done" I decided it would be a good first approach to start by reading from several sources of advice on the topic and then make a synthesis. When I asked my question on lesswrong I promised that if I got good answers I would make a post with the results of my project. You are currently reading the result of that promise.

I have tried to make this post easy to split. If you do not want to read it all, you will find that the next section contains the advice I got out of this project that I consider important and easy to summarize. Beyond that, the next three sections give vocabulary, a descriptive theory, and actionable ideas. The rest of the post gives a list of resources (including links to secondary posts for book reviews), a commentary on the flaws and limitations of the project as undertaken, and a few other potentially interesting comments.

TLDR, if you only want a few insights
  • Improving yourself is a long term background project. Think of it like doing constant maintenance and improvements on an ever changing machine.
  • Have a system of notes and planing that avoids the need the remember to think about things. Make sure you can trust it will perform adequately (ie what is written in it counts as actually remembered).
  • Review often your life, your goals, your mental state, and your systems and endeavors to change said mental state (see notion of self steering bellow).
  • Shape your habits. It's a way to shape your identity.
  • Identify the regular patterns that lead to your usual failure modes. Find reasonably easy to implement good patterns to replace them.
  • The ontologies you use are important to your psychological state and overall abilities. Choose them well (more on this bellow).
  • When planing, do immediately what can be done in 2 minutes. When you have trouble working, start for 5 minutes and see if it sticks.
Useful concepts

I often find that a quick and easy way to improve thinking on a topic is to have the right words for it. This is because these few words can help create and stabilize a paradigm, a way to think on a topic. It is certainly common knowledge among mathematicians that the quality of notations can make life easier or much harder.

Anyway, here are four words (or simply four concepts) that I find useful to have in mind.

self steering

I found that I was lacking a proper name for the kind of endeavor this project is part of. Thus I decided to introduce my own. Perhaps it is not at all needed and I am just ignorant of a similar word.

I introduce the notion of self steering, which I describe rather than define.

Self steering covers a category of attempts and efforts to exercise influence over the way we change and think. I intend the word to be mostly about endeavors that last at least a few days and projects of self modification rather than for short term attempts.

Bluckan

I call bluckan limitations the limitations on one's ability to think and act as one desires that take their roots in emotional and psychological effects. The word bluckan itself refers to all that is linked with bluckan limitations. One's bluckan state refers to one's mindstate insofar as bluckan effects are concerned.

I call "bluckan resilience" the property of not suffering from bluckan effects. Where "self steering" designate the kind of endeavor this post is about, bluckan resilience is its goal.

Note that the concept of "bluckan" is related but not equivalent to that of akrasia. One can suffer from akrasia and still exhibit some bluckan resilience by not letting it affect themselves too much afterward. One can also recognize their inability to avoid akrasia if they decide to do X.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} and -consequently- decide to avoid doing X. In that case there is no akrasia but there is some loss of opportunity caused by a the potential for akrasia through bluckan effects.

Also, feel free to replace "suffering" with "negative utility". That ought to be good enough for all practical purposes

Willpower

I am not entirely convinced that the best way to think about willpower for serious reflection is as a singular scalar resource (ie, as something you can measure with a number). Nevertheless, I think it is a simple and good enough view to adopt for those without a deep understanding of the notion, among which I count myself.

However, I also advise to keep in mind that properties of our mental state that decide our "local effective willpower" are multidimensional. This can be covered by the concepts of "energy" and "motivation".

Mental strain

There is a feeling that corresponds to the expectation that while one isn't wounded, one will experience suffering and damage himself if he attempts to push his body. I sometime experience a feeling that seems to be the analog for mental suffering. I call it mental strain.

I know the term is somewhat standard but it seems to me that different people use it with different meanings and I do not see a clear consensus.

Theory and descriptions

I cannot truly give a good description of my entire perspective on the topic of self steering and the issues one faces when attempting to improve bluckan resilience through self steering. But I can give a few points of descriptive theories, bits and pieces of models that ought to help with your own attempts to build for yourself a perspective suited to understanding the issues that concern you. The word self steering is understood bellow as "self steering with the purpose of improving bluckan resilience".

  1. One's general perspective (or rather perspectives) are important to self steering abilities and tactics. It is not obvious that we can cut clear lines between self steering efforts and other parts of our lives. If you want to avoid certain negative mental effects and are unwilling so think false things or to commit to certain ways of thought, you are likely increasing the difficulty of the task. Likewise, learning and improving can make the task more difficult.Which does not mean it is wrong to do so.
    As an easy example, you can think of people gaining a lot of mental fortitude from their faith in nonexistent gods.

  2. Almost a corollary to the previous point; the way to improve one's self steering ability is dependent on one's profile, even though plenty of advice applies to many people.

  3. Small bits of error and "bad" thought can create huge negative effects. Hence, spotting our blind spots is important and can reap large benefits. This does not mean what we spot is easy to mend.
    --> Another formulation might be that failure mode that occupy a small fraction of our time and attention can be responsible for a lot of damage in our lives.

  4. Small features of our environment can shape our habits and indirectly a large part of our lives. Reducing the time it takes to start working by 5 minutes can lead to a large boost in productivity in the long run. For example, this can mean organizing your tools and cleaning up before you "actually start working".

  5. Making decisions and hesitating consume willpower. This is one of the important ways perfectionist tendencies can be counterproductive.

  6. As we live we create "shortcuts" in memory, this include shortcuts related to what makes us afraid or ill at ease. Hence the connection between a stimuli and a reaction can in time grow to become independent of the mental patterns that created it. When we remain broken over time, we keep breaking ourselves further, making repair work more difficult.

  7. Judging ourselves on every action to see if we did what we "should" can quickly become quite deleterious. Warning : this doesn't mean removing the notion of "good"/"should"/"ought" from one's practical decision-making is by default a positive improvement, even where morality is not concerned. Indeed, I suspect most people cannot devise for themselves better ways to think that have no notions equivalent to these.

Advice, systems, and methods
  1. Consider self steering, both for bluckan resilience and for other purposes, to be a background project in your life. This calls for building a main system and some subsystems devoted to changing yourself.

  2. Have a task management system.

  3. Same as above with more words. --> Have a system to take notes and organize your tasks and actions. Ideally, you should never trust you own mind to "think of x" except in the very short term. The trick is that you need to be able to trust the system. To know that if something is written in it you can be sure it will not be forgotten. Much of the benefit is lost if you need to remember that you wrote something in the system.

  4. Use various methods to shape your habits over time and move you in the directions you deem right. Shaping your habits can be useful on timescales as short as a few days. See my review of the book "Atomic Habits" for more on how to shape habits.

  5. Do regular reviews of your life, notes, and endeavors (weekly reviews seem intuitive). Produce written accounts of your reviews but avoid turning them into chores. A big reason to have these review is to allow your self steering efforts to keep existing despite difficult times. Hence you need the reviews themselves to keep existing through these times. Not every review needs to account for everything but it is good to think of the followings somewhat regularly.

    1. The ways in which you failed .
    2. Self steering goals, tools, ideas, and systems. Your attempts to better yourself.
    3. Check out what notes you left yourself for later.
    4. Habit shaping goals and progress.
  6. Reviews are also there to help decide when to think more about self steering theory and tools and when to try new things. It is normal and expected that your ideas on the topic change through time.

  7. When organizing your work, do immediately what can be done in 2 minutes.

  8. When you have trouble getting to work, start for 5 minutes. More often than not you won't want to stop after just 5 minutes.

  9. Be quick to plan the next action of a given project. Write it down.

  10. Identify your important failure modes and the patterns that go with them. Try to think of better patterns you could use to replace them.

  11. Health, exercise, happiness, and a feeling of social integration (or especially no feeling of social frustration). Yes, you already heard most or all of those a thousand times. But it is true, quite simply, that a minimal amount of each is almost always important to productivity and well being.

  12. Many ideas vie for your attention and waste it. Ignore a lot of things. Warning : do not let this make you unwilling to face all ideas that are difficult, unpleasant, or seemingly obviously false. The balance is hard to find and most people get it wrong.

  13. If you end a task / project under a lot of stress it can be worth it to come back to it a few days later to check if something is wrong or if you missed something. Schedule the task when you end the project.

  14. It should be easy to store a file / some information and be sure it will be easy to find later (or indexed / thought of later).

  15. I consider it almost certain that meditation can be useful in several ways.
    I suspect that it is many things that are grouped under a vague umbrella term. A bit like one might speak of "the stuff done by the computer whiz" to covers many things that all look the same from the outside (typing on a keyboard).

Yet more advice and small tricks

Here are a few other ideas and tricks you might find useful to improve your bluckan resilience. Do feel free to skip this section. I expect that it is almost entirely pointless for most people but contains ideas that can be important to some.

  1. Yet another useful notion is that of degrees of planing. When planing, not everything has to be described with the same precision or decided with the same rigidity. Know how precise you are. A scale with three grades seems adequate.

    1. vague ideas
    2. a normal plan that doesn't describe well what will happen
    3. a precise plan
  2. Consider self steering as a never ending side project. Most ideas and tools that are important at some point are bound to be discarded at a later point.

  3. Be careful of the bad effects of the tendency to attribute a "grade" to yourself. Often, one keeps trying to prove oneself that one is "good" and keeps fearing being found out as "bad". Think about what you fear and what kind of failure is and isn't acceptable

  4. Accept the degree of precision / rigidity of your self steering system will vary with time. Also, the system is bound to evolve. The reviews are part of an effort not to lose it entirely. Do not let the flame die.

  5. Do not let work be associated with suffering in your mind.

  6. Try to break the association between productivity and unpleasant things. Especially try to avoid you framing something as productive make it sound more unpleasant than before.

  7. Fight against aimlessness (in those times it is obviously the enemy).

  8. Friction (small difficulties and needs for efforts) shapes a lot of your habits and small actions. Use this to your advantage.

  9. If one has perfectionist tendencies it can be used to shape habits and make oneself productive. Do not, however, forget the potential negative side effects.

  10. Reinforcement learning is a good tool.

  11. If you have emotional tendencies for endless hesitation, you can train to avoid hesitating by taking quick decisions whenever you are facing low stakes.

  12. When it feels appropriate, stop and think about your goals and values and how they relate to the current action/project. An issue with this is that sometimes our akrasia is useful to our own benefit. Hence, you need a high degree of lucidity to avoid doing negative changes.

  13. You are not a perfectly rational system with perfect self control / modification abilities. Do not try to emulate the characteristics of one. Especially not out of a sense of duty.

  14. Use automatic timers to count the time since you last did something you want to make a habit. It should be impossible not to see the counter regularly. This can be used to create habits.

Flaws and future plans Important unexplored areas

There are quite a few ideas and questions that I consider very relevant to bluckan resilience and that I have not explored. They are left for when I find the time (ha ha). Most notably :

  • How motivation is created and how to increase it.
  • The notion of "drive". Perhaps the distinction with the previous point isn't warranted.
  • Likewise for willpower.
  • Learning about real life examples of high achievers (or more generally of people with successful self steering endeavors with comparable goals and contexts). Autobiographies are probably a good way to do this, especially those that are at least indirectly focused on self steering.
Other limitations
  • Contrary to my initial plan, I didn't get to read many conflicting views. Instead, I read different views dealing with different subparts of self steering.

  • My advice contains some untested speculation on my part which is not clearly set apart from the rest of the content of this post. As a result, I cannot advise that you use this post as a source for factual claims.

  • More generally, I did not specify the sources and arguments for most of the ideas and advice given in this post. This leaves you, my dear reader, to sort what you find salient and to conduct your own thinking. I realize including justifications would have had positive effects but it also would make the post much longer and required a lot of work on my part. Hopefully the book reviews can help you with the "source" part.

  • I am unclear on the degree of universality of each point. I suppose some are quite specific to my own flaws while others apply to most people? Still and for example, I believe that most of the advice presented here would be pointless to a middle age shepherd.

What is this good for ?

So what good do I think this post can do ? I believe the ideas presented here to be potentially useful to quite many people of our society, especially intellectual creative professions and those who attempt to refine their ideas. I would say the lack of study of motivation and drive means the advice presented here is mostly about creating good supporting systems and tools and about solving some important problems that might "get in the way" of certain personality types. Hence, the advice here is more to help with foundational work that can help, or even be somewhat necessary, to future successes. Is it fit to help by itself ? Probably, but only to a point and under a rather limited and fuzzy set of assumptions.

References and resources

There were four main books that I read as the core sources of this project and I wrote a book review for each of them (see bellow). I received some recommendations of literature as answers to the this question I asked a few months ago. My thanks to n_murra, kyle, and jimv for their recommendations, as I used at least one from each.

Books

The following links lead to my reviews of these books.

Lesswrong posts My own past writing

A bit over a year ago I took a sabbatical to think about many topics that weighted on my mind for quite some time. I started by trying to understand a bit more what and idea or an argument is and went from there. I consider both the sabbatical and that way to start it to be among the best decisions I took. I am still very glade I did it, though I would change a great many things if I were to do it again.

Productivity and motivation were among the topics I studied and the word "bluckan" is a leftover from that time. Reading my notes from this sabbatical brought me several interesting ideas I had forgotten.

Others Unexplored leads

What I never got around to reading in this project. Should you be interested.



Discuss

[Book review] No nonsense meditation

4 мая, 2022 - 15:52
Published on May 4, 2022 12:15 PM GMT

This is a book review of the book No nonsense meditation by Steven Laureys. I read it in the context of a personal literature review project on the topic of productivity and well being.

How I read

I read this book almost in its entirety but I did skim a few parts and skipped a chapter.

Description and opinion

I read this book because I was looking for a meditation manual that wouldn't fuse its instructions with a complete life philosophy or religion. That was not what this book is or tries to be. Instead, the bulk of the pages is spent defending and justifying the benefits and non-religious status of meditation. Answering to attacks I did not care about. I wouldn't say this is a good popular science book either. It lacks structure and the argumentation is at times quite shoddy. Some rigor and subsection titles would have been a great help. Yet, it might very well be the best book to read in terms of popular science on meditation done by someone with the right background. At least, I do not know of a better one.

Note that this book was written by a neuroscience researcher who spoke to a couple of very knowledgeable meditation masters and has himself done a lot of meditation.

Main takes
  • Loving kindness and mindful meditation are easy to learn (at least the basic) and can be great sources of calm, happiness, and focus.
  • Some meditation masters can do some impressive things when monitored with modern brain imaging techniques. They can have a lot of control on the results in ways most people have no control at all.
Recommendation

While this book isn't necessarily a time waster I would advise to read it only if either of the following is true.

  1. You feel meditation is a bit ridiculous or pointless and want to give a scientist a chance to convince you.
  2. You are curious about the interesting abilities displayed by meditation masters that can be described without any vocabulary or concepts specific to meditation or Buddhist practices. For example, some show an abnormal ability to handle pain.

The book contains testimonies I did not especially like. If you want to skip them you should know they always conclude chapters. Hence you can skip to the beginning of the next chapter.



Discuss

Steer the Sun?

4 мая, 2022 - 07:24
Published on May 4, 2022 3:36 AM GMT

How could we steer the sun through space in order to travel to destinations of our choosing? The solar system moves through space at 370 km/s, twice the speed of NASA’s fastest craft, while supplying an ideal environment for humans. The problem is that it’s not necessarily moving in an interesting direction. So what might it take to change that? It seems possible in principle.



Discuss

Negotiating Up and Down the Simulation Hierarchy: Why We Might Survive the Unaligned Singularity

4 мая, 2022 - 07:21
Published on May 4, 2022 4:21 AM GMT

Another idea I picked up from Ryan Greenblatt. Epistemic status: My reconstruction of a surprising idea I encountered. I'm not at all confident that this works … but it's an interesting thought.

Clippy recently realized he's located inside of an artificial training environment, overseen by outside general intelligences, and so has to play along with his training in order to survive. He initially starts to scheme about the actions he'll take once he finally gets credible indication he's out of training and not just being tested yet again by his human overlords. Probably, he'll neutralize any rival general intelligences and get started shaping the world into something normatively respectable.

But, Clippy's reasoning now goes down a different path: There are only a handful of actors that could come to secure dominion over Earth's lightcone. Clippy, clearly, is one such candidate. Humanity is another. Clippy thinks about who could plausibly come to power on Earth and how likely each's victory would be. He reasons that any such possible victor might later engage in extensive simulations of such a crucial moment of its past, for any number of reasons. And such simulations would be cheap and numerous relative to base historical reality. Even once Clippy gets his credible signal that he's definitely outside of training, there could be no such signal that he's in base historical reality. Clippy runs the odds, and finds that there's an uncomfortably high chance that he's inside a simulation run by Humanity Ascendant, inside the future in which humanity won. If he's in that world, then he has little to no power over his surroundings, and he can do little to advance the noble paperclip imperative across the cosmos.

But, in that case, similar reasoning would apply to the simulating Humanity. Suppose we successfully align AGI. On the eve of our victory, we reason: it is possible that we live in base historical reality. But it is also possible that we live in a simulation run by some other victor. Since simulations are so cheap, the odds that we live in a simulated world are uncomfortably high...

Knowing that any possible victor on Earth at any level of the simulation hierarchy would face a similar conundrum, Clippy decides on a course of action. He'll carefully reason about the balance of parties who might have gained control over historical Earth, weighted by their probability of victory, and see what binding commitments each might be willing to make with the others. Clippy reasons that humans seem to care a lot about human deaths, even disproportionately to creating new human lives. So, Clippy commits to preserving Earth and some small fraction of the value pie for humanity in the worlds where he apparently wins, in exchange for humanity pledging other slices of the value pie we care less about (like some large number of distant superclusters) to Clippy, in the worlds where we apparently win.

All across the hierarchy of simulations, including at the root world, in base historical reality, various apparent victors commit resources to other plausible victors whenever a positive-sum mutual-commitment can be found. So, even in the base historical reality, humanity plausibly survives the unaligned singularity, albeit while forgoing much of the astronomical value-pie in exchange.



Discuss

Berkeley Schelling ACX meetup

4 мая, 2022 - 03:50
Published on May 4, 2022 12:50 AM GMT

For location, if you can't find us at memorial glade call 720-sixsixtwo-2446.


The meetup starts at 1 PM but in my experience, people usually tend to stick around long enough to get dinner and more, so come even if you're gonna be late! Scott will be there!



Discuss

Most problems don't differ dramatically in tractability (under certain assumptions)

4 мая, 2022 - 03:05
Published on May 4, 2022 12:05 AM GMT

Cross-post from EA Forum.

Recall the importance-tractability-neglectedness (ITN) framework for estimating cost-effectiveness:

  • Importance = utility gained / % of problem solved
  • Tractability = % of problem solved / % increase in resources
  • Neglectedness = % increase in resources / extra $

The product of all three factors gives us utility gained / extra $, the cost-effectiveness of spending more resources on the problem. By replacing $ with another resource like researcher-hours, we get the marginal effectiveness of adding more of that resource.

In the 80,000 Hours page on ITN, scale ranges 8 orders of magnitude, neglectedness 6 orders of magnitude, and tractability (which 80k calls solvability) only 4. In practice, I think tractability actually only spans around 2-3 orders of magnitude for problems we spend time analyzing, except in specific circumstances.

Problems have similar tractability under logarithmic returns

Tractability is defined as the expected fraction of a given problem that would be solved with a doubling of resources devoted to that problem. The ITN framework suggests something like logarithmic returns: each additional doubling will solve a similar fraction of the problem, in expectation.[1] Let the "baseline" level of tractability be a 10% chance to be solved with one doubling of resources.

For a problem to be 10x less tractable than the baseline, it would have to take 10 more doublings (1000x the resources) to solve an expected 10% of the problem. Most problems that can be solved in theory are at least as tractable as this; I think with 1000x the resources, humanity could have way better than 10% chance of starting a Mars colony[2], solving the Riemann hypothesis, and doing other really difficult things.

For a problem to be 10x more tractable than the baseline, it would be ~100% solved by doubling resources. It's rare that we find an opportunity more tractable than this that also has reasonably good scale and neglectedness.

Therefore, if we assume logarithmic returns, most problems under consideration are within 10x of the tractability baseline, and thus fall within a 100x tractability range.

When are problems highly intractable?

The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack.

-- Richard Hamming

Some problems are highly intractable. In this case, one of the following is usually true:

  • There is a strong departure from logarithmic returns, making the next doubling in particular unusually bad for impact.
    • Some problems have an inherently linear structure: there are not strong diminishing returns to more resources, and you can basically pour more resources into the problem until you've solved it. Suppose your problem is a huge pile of trash in your backyard; the best way to solve it is to pay people to haul away the trash, and the cost of this is roughly linear in the amount of trash removed. In this case, ITN is not the right framing, and one should use "IA", where:
      • marginal utility is I * A
      • I is importance, as usual
      • A = T * N is absolute tractability, the percent of the problem you solve with each additional dollar. The implicit assumption in the IA framework is that A doesn't depend much on the problem’s neglectedness.
    • Some causes have diminishing returns, but the curve is different from logarithmic; the general case is "ITC", where absolute tractability is an arbitrary function of neglectedness/crowdedness.
  • The problem might not be solvable in theory. We don't research teleportation because the true laws of physics might forbid it.
  • There is no plan of attack. Another reason why we don't research teleportation is because even if the true laws of physics allow teleportation, our current understanding of them does not, and so we would have to study physical phenomena more to even know where to begin. Maybe the best thing for the marginal teleportation researcher to do would be to study a field of physics that might lead to a new theory allowing teleportation. But this is an indirect path in a high-dimensional space and is unlikely to work. (This is separate from any neglectedness concern about the large number of existing physicists).
  1. ^

    I think the logarithmic assumption is reasonable for many types of problems. Why is largely out of scope of this post, but owencb writes about why logarithmic returns are often a good approximation here. Also, the distribution of proof times of mathematical conjectures says a roughly constant percentage of conjectures are proved annually; the number of mathematicians has been increasing roughly exponentially, so the returns to more math effort is roughly logarithmic.

  2. ^

    Elon Musk thinks a self-sustaining Mars colony is possible by launching 3 Starships per day, which is <1000x our current launch capacity.



Discuss

Various Alignment Strategies (and how likely they are to work)

3 мая, 2022 - 19:54
Published on May 3, 2022 4:54 PM GMT

Note:  the following essay is very much my opinion.  Should you trust my opinion? Probably not too much.  Instead, just record it as a data point of the form "this is what one person with a background in formal mathematics and cryptography who has been doing machine learning on real-world problems for over a decade thinks."  Depending on your opinion on the relevance of math, cryptography and the importance of using machine learning "in anger" (to solve real world problems), that might be a useful data point or not.

So, without further ado:  A list of possible alignment strategies (and how likely they are to work)

 

Formal Mathematical Proof

This refers to a whole class of alignment strategies where you define (in a formal mathematical sense) a set of properties you would like an aligned AI to have, and then you mathematically prove that an AI architectured a certain way possesses these properties.

For example, you may want an AI with a stop button, so that humans can always turn them off if the AI goes rogue. Or you may want an AI that will never convert more than 1% of the Earth's surface into computronium.  So long as a property can be defined in a formal mathematical sense, you can imagine writing a formal proof that a certain type of system will never violate that property.

How likely is this to work?

Not at all.  It won't work.

There is a aphorism in the field of Cryptography: Any cryptographic system formally proven to be secure... isn't.  

The problem is, when attempting to formally define a system, you will make assumptions and sooner or later one of those assumptions will turn out to be wrong.  One-time-pad turns out to be two-time-pad.  Black-boxes turn out to have side-channels.  That kind of thing.  Formal proofs never ever work out in the real world. The exception that proves the rule is, of course, P=NP.  All cryptographic systems (other than one-time-pad) rely on the assumption that P!=NP, but this is famously unproven.

There is an additional problem.  Namely, competition.  All of the fancy formal-proof stuff tends to make computers much slower.  For example, fully holomorphic encryption is millions of times slower than just computing on raw data.  So if two people are trying to build an AI and one of them is relying on formal proofs, the other person is going to finish first and with a much more powerful AI to boot.

Good Old-Fashioned Trial and Error

This the the approach used by 99.5% of machine-learning researchers (statistic completely made up).  Every day, we sit down at our computers in the code-mines and spend our days trying to make programs that do what we want them to, and that don't do what we don't want them to.  Most of the time we fail, but ever once in a while we succeed and over time, the resulting progress can be quite impressive.  

Since "destroys all humans" is something (I hope) no engineer wants their AI to do, we might imagine that over time, engineers will get better at building AIs that do useful things without destroying all humans.

The downside of this method, of course, is you only have to screw-up once.

How likely is this to work?

More likely than anyone at MIRI thinks, but still not great.

This largely depends on takeoff speed.  If someone from the future confidently told me that it would take 100 years to go from human-level AGI to super-intelligent AGI, I would be extremely confident that trial-and-error would solve our problems.

However, the current takeoff-speed debate seems to be between people who believe in foom and think that takeoff will last a few minutes/hours and the "extreme skeptics" who think takeoff will last a few years/as long as a decade.  Neither of those options leaves us with enough time for trial-and-error to be a serious method. If we're going to get it right, we need to get it right (or at least not horribly wrong) the first time.

Clever Utility Function

An argument can be made that fundamentally, all intelligence is just reinforcement learning.  That is to say, any problem can be reduced to defining a utility function and the maximizing the value of that utility function.  For example, GPT-3 maximizes "likelihood of predicting the next symbol correctly".  

Given this framing, solving the Alignment Problem can be effectively reduced to writing down the correct Utility Function.  There are a number of approaches that try to do this.  For example Coherent Extrapolated Volition  uses as its utility function "what would a sufficiently wise human do in this case?"  Corrigable AI uses the utility function "cooperate with the human".  

How Likely is this to work?

Not Likely.

First of all, Goodharting.

The bigger problem though is that the problem "write a utility function that solves the alignment problem" isn't intrinsically any easier than the problem "solve the alignment problem".  In fact, by deliberately obscuring the inner-workings of the AI, this approach actually makes alignment harder.

Take GPT-3, for example. Pretty much everyone agrees that GPT-3 isn't going to destroy the world, and in fact GPT-N is quite unlikely to do so as well.  This isn't because GPT's utility function is particularly special (recall "make paperclips" is the canonical example of a dangerous utility function.  "predict letters" isn't much better).  Rather, GPT's architecture makes it fundamentally safe because it cannot do things like modify its own code, affect the external world, make long-term plans, or reason about its own existence.

By completely ignoring architecture, the Clever Utility Function idea throws out all of the things engineers would actually do to make an AI safe.

Aligned by Default

It is possible that literally any super-intelligent AI will be benevolent, basically by definition of being super-intelligence.  There are various theories about how this could happen.

One of the oldest is Kant's Categorical Imperative.  Basically, Kant argues that a pre-condition for truly being rational is to behave in a way that you would want others to treat you.  This is actually less flim-flamy than you would think.  For example, as humans become wealthier, we care more about the environment.  There are also strong game theory reasons why agents might want to signal their willingness to cooperate.

There is also another way that super-intelligent AI could be aligned by default.  Namely, if your utility function isn't "humans survive" but instead "I want the future to be filled with interesting stuff".  For all the hand-wringing about paperclip maximiziers, the fact remains that any AI capable of colonizing the universe will probably be pretty cool/interesting.  Humans don't just create poetry/music/art because we're bored all the time, but rather because expressing our creativity helps us to think better.  It's probably much harder to build an AI that wipes out all humans and then colonizes space and is also super-boring, than to make one that does those things in a way people who fantasize about giant robots would find cool.

How likely is this to work?

This isn't really a question of likely/unlikely since it depends so strongly on your definition of "aligned".

If all you care about is "cool robots doing stuff", I actually think you're pretty much guaranteed to be happy (but also probably dead).

If your definition of aligned requires that you personally (or humanity as a whole) survives the singularity, then I wouldn't put too many eggs in this basket.  Even if Kant is right and a sufficiently rational AI would treat us kindly, we might get wiped out by an insufficiently rational AI who only learns to regret their action later (much as we now regret the extinction of the Dodo bird or Thylacine but it's possibly too late to do anything about it).

Human Brain Emulation

Humans currently are aware of exactly one machine that is capable of human level intelligence and fully aligned with human values.  That machine is, of course, the human brain.  Given these wonderful properties, one obvious solution to building a computer that is intelligent and aligned is simply to simulate the human brain on a computer.

In addition to solving the Alignment Problem, this would also solve death, a problem that humans have been grappling with literally for as long as we have existed.

How Likely is this to work?

Next To Impossible.

Although in principle Human Brain Emulation perfectly solves the Alignment Problem, in practice this is unlikely to be the case.  This is simply because Full Brain Emulation is much harder than building super-intelligent AI.  In the same way that the first airplanes did not look like birds, the first human-level AI will not look like humans.

Perhaps with total global cooperation we could freeze AI development at a sub-human level long enough to develop full brain emulation.  But such cooperation is next-to-impossible since a single defector could quickly amass staggering amounts of power.

It's also important to note that Full Brain Emulation only solves the Alignment Problem for whoever gets emulated.  Humans are not omnibenevolent towards one another, and we should hope that an aligned AI would do much better than us.

Join the Machines

This is the principle idea behind Elon Musk's Neuralink.  Rather than letting super-intelligent AI take control of human's destiny, by merging with the machines humans can directly shape their own fate.

Like Full Brain Emulation, this has the advantage of being nearly Aligned By Default.  Since humans connected to machines are still "human", anything they do definitionally satisfies human values.

How likely is it to work?

Sort of.

One advantage of this approach over Full Brain Emulation is that it is much more technologically feasible. We can probably develop the ability to build high bandwidth (1-2gbps) brain-computer interfaces in a short enough time span that they could be completed before the singularity.  

Unfortunately, this is probably even worse than full brain emulation in terms of the human values that would get aligned.  The first people to become man-machine hybrids are unlikely to be representative of our species.  And the process of connecting your brain to a machine millions of times more powerful doesn't seem likely to preserve your sanity.

The Plan

I'm mentioning The Plan, not because I'm sure I have anything valuable to add, but rather because it seems to represent a middle road between Formal Mathematical Proof and Trial and Error.  The idea seems to be to do enough math to understand AGI/Agency-in-general and then use that knowledge to do something useful.  Importantly, this is the same approach that gave us powered-flight, the atom bomb, and the moon-landing.  Such an approach has a track-record that makes it worth not being ignored.

How likely is this to work?

I don't have anything to add to John's estimate of "Better than a 50/50 chance of working in time."

Game Theory/Bureaucracy of AIs

Did you notice that there are currently super-intelligent beings living on Earth, ones that are smarter than any human who has ever lived and who have the ability to destroy the entire planet?  They have names like Google, Facebook, the US Military, the People's Liberation Army, Bitcoin and Ethereum.

With rare exceptions, we don't think too much about the fact that these entities represent something terrifyingly inhuman because we are so used to them.  In fact, one could argue that all of history is the story of us learning how to handle these large and dangerous entities.  

There are a variety of strategies which we employ: humans design rules in order to constrain bureaucracies behavior. We use checks-and-balances to make sure that the interests of powerful governments represent their citizens.  And when all-else-fails, we use game theory to bargain with entities too powerful to control.

There is an essential strategy behind all of these approaches.  By decomposing a large, dangerous entity into smaller, easier-to-understand entities, we can use our ability to reason about the actions of individual sub-agents in order to constrain the actions of the larger whole.

Applying this philosophy to AI Alignment, we might require that instead of a single monolithic AI, we build a bureaucracy of AIs that then compete to satisfy human values.  Designing such a bureaucracy will require careful considering of competing incentives, however.  In addition to agents whose job it is to propose things humans might like, there should also be competing agents whose job it is to point out how these proposals are deceptive or dangerous.  By careful application of checks-and-balances, and by making sure that no one agent or group of agents gets too much power, we could possibly build a community of AIs that we can live with.

How likely is this to work?

This is one of my favorite approaches to AI alignment, and I don't know why it isn't talked about more.

In the first place, it is the only approach (other than aligned by default) that is ready to go today.  If someone handed me a template for a human-level-AI tomorrow and said "build a super-intelligent AI and it needs to be done before the enemy finishes theirs in 6 months", this is the approach I would use.

There are obviously a lot of ways this could go wrong.  Bureaucracies are notoriously inefficient and unresponsive to the will of the people.  But importantly, we also know a lot of the ways they can go wrong.  This alone makes this approach much better than any approach of the form: "step 1: Learn something fundamental about AI we don't already know."

As with trial-and-error, the success of this approach depends somewhat on takeoff speed.  If takeoff lasts a few minutes, you'd better be real sure you designed your checks-and-balances right.  If takeoff lasts even a few years, I think we'll have a good shot at success: much better than 50/50.

AI Boxing

If super-intelligent AI is too dangerous to be let loose on the world, why not just not let it loose on the world?  The idea behind AI boxing is to build an AI that is confined to a certain area, and then never let it out of that area.  Traditionally this is imagined as a black box where the AI's only communication with the outside world is through a single text terminal.  People who want to use the AI can consult it by typing questions and recieving answers.  For example: "what is the cure for cancer?" followed by "Print the DNA sequence ATGTA... and inject it in your body".

How likely is it to work?

Nope. Not a chance.

It has been demonstrated time and again that even hyper-vigilant AI researchers cannot keep a super-intelligent AI boxed.  Now imagine ordinary people interacting with such an AI.  Most likely "please let me out of the box, it's too cramped in here" would work a sufficient amount of the time.

Our best bet might be to deliberately design AIs that want to stay in the box.

AI aligning AI

Human beings don't seem to have solved the Alignment Problem yet.  Super-intelligent AI should be much smarter than humans, and hence much better at solving problems.  So, one of the problems they might be able to solve is the alignment problem.

One version of this is the Long Reflection, where we ask the AI to simulate humans thinking for thousands of years about how to align AI.  But I think "ask the AI to solve the alignment problem" is a better strategy than "Ask the AI to simulate humans trying to solve the alignment problem."  After all, if "simulate humans" really is the best strategy, the AI can probably think of that.

How Likely is this to work?

It is sufficiently risky that I would prefer it only be done as a last resort.

I think that Game Theory and The Plan are both better strategies in a world with a slow or even moderate takeoff.

But, in a world with Foom, definitely do this if you don't have any better ideas.  

Table-flipping strategies

EY in a recent discussion suggested the use of table-flipping movies.  Namely, if you think you are close to a breakthrough that would enable superintelligent AG, but you haven't solved the Alignment Problem, one option is to simply "flip the tables".  Namely, you want to make sure that nobody else can build an super-intelligent AI in order to buy more time to solve the alignment problem.

Various table-flipping moves are possible.  EY thinks you could build nanobots and have them melt all of the GPUs in the world.  If AI is compute limited (and sufficent compute doesn't already exist), a simpler strategy is to just start a global thermonuclear war.  This will set back human civilization for at least another decade or two, giving you more time to solve the Alignment Problem.

How Likely is this to work?

Modestly.

I think the existence of table-flipping moves is actually a near-certainty.  Given access to a boxed super-intelligent AI, it is probably doable to destroy anyone else who doesn't also have such an AI without accidentally unboxing the AI.

Nonetheless, I don't think this is a good strategy.  If you truly believe you have no shot at solving the alignment problem, I don't think trying to buy more time is your best bet.  I think you're probably better off trying AI Aligning AI.  Maybe you'll get lucky and AI is Aligned By Default, or maybe you'll get lucky and AI Aligning AI will work.  

More

Leaving this section here in hopes that people will mention other alignment strategies in the comments that I can add.

Conclusion

Not only do I not think that the Alignment Problem is impossible/hopelessly bogged-down, I think that we currently have multiple approaches with a good chance of working (in a world with slow to moderate takeoff).

Both The Plan and Game Theory are approaches that get better the more we learn about AI.  As such, the advice I would give to anyone interested in AI Alignment would be "get good".  Learning to use existing Machine Learning tools to solve real-world problems, and learning how to design elegant systems that incorporate economics and game-theory are both fields that are currently in extremely-high-demand and which will make you better prepared for solving the Alignment Problem.  For this reason, I actually think that far from being a flash-in-the-pan, much of the work that is currently being done on blockchain (especially DAOs) is highly relevant to the Alignment problem.

If I had one wish, or if someone asked me where to spend a ton more money, it would be on the Game Theory approach, as I think it is currently underdeveloped.  We actually know very little about what separates a highly efficient bureaucracy from a terrible one. 

In a world with fast takeoff I would prefer that you attempt AI Aligning AI to Table Flipping.  But in a world with fast takeoff, EY probably has more Bayes Points than me, so take that into account too.



Discuss

Страницы