« Banning Bad News | Main | Can Counterfactuals Be True? »

July 23, 2008


Hardly the most profound addendum, I know, but dummy numbers *can* be useful for illustrative purposes - for instance, to show how steeply probabilities decline as claims are conjoined.

If I could prevent only one of these events, I would prevent the lottery.

I'm assuming that this is in a world where there are no payoffs to the LHC; we could imagine a world in which it's decided that switching the LHC on is too risky, but before it is mothballed a group of rogue physicists try to do the riskiest experiment they can think of on it out of sheer ennui.

In the sentence "Trying to catch a flying ball, you're probably better off with your brain's built-in mechanisms, then using deliberative verbal reasoning to invent or manipulate probabilities," I think you meant "than" rather than "then"?

great post.

This is mostly what economists refer to as the difference between implicit and explicit knowledge. The difference between skills and verbal knowledge. I strongly recommend Thomas Sowell's "Knowledge and Decisions".

If all you have is a gut feeling of uncertainty, then you should probably stick with those algorithms that make use of gut feelings of uncertainty, because your built-in algorithms may do better than your clumsy attempts to put things into words.

I would like to add something to this. Your gut feeling is of course the sum of experience you have had in this life plus your evolutionary heritage. This may not be verbalized because your gut feeling (as an example) also includes single neurons firing which don't necessarily contribute to the stability of a concept in your mind.

But I warn against then simply following one's gut feeling; of course, if you have to decide immediately (in an emergency), there is no alternative. Do it! You can't get better than the sum of your experience in that moment.

But usually only having a gut feeling and not being able to verbalize should mean one thing for you:
Go out and gather more information! (Read books to stabilize or create concepts in your mind; do experiments; etc etc)

You will find that gut feelings can change quite dramatically after reading a good book on a subject. So why should you trust them if you have the time to do something about them, viz. transfer them into the symbol space of your mind so the concepts are available for higher-order reasoning?

Calculating probabilities about nearly any real world event is extremely complex. Someone who accepts the logic of your post shouldn't believe there is much value to Bayesian analysis other then allowing you to determine whether new information should cause you to increase or decrease your estimate of the probability of some event occurring.

It should be possible for someone to answer the following question: Is the probability of X occurring greater or less than Y? And if you answer enough of these questions you can basically determine the probability of X.

What's the probability the LHC will save the world? That either some side effect of running it, or some knowledge gained from it, will prevent a future catastrophe? At least of the same order of fuzzy small non-zeroness as the doomsday scenario.

I think that's the larger fault here. You don't just have to show that X has some chance of being bad in order to justify being against it, you also have to show it it's predictably worse than not-X. If you can't, then the uncertain badness is better read as noise at the straining limit of your ability to predict - and that to me adds back up to normality.

For example, I would be substantially more alarmed about a lottery device with a well-defined chance of 1 in 1,000,000 of destroying the world, than I am about the Large Hadron Collider switched on. If I could prevent only one of these events, I would prevent the lottery.

On the other hand, if you asked me whether I could make one million statements of authority equal to "The Large Hadron Collider will not destroy the world", and be wrong, on average, around once, then I would have to say no.

Hmm... might this be the heuristic that makes people prefer a 1% chance of 1000 deaths to a definite death for 5? The lottery would definately destroy worlds, with as many deaths as killing over six thousand people in each Everett branch. Running the LHC means a higher expected number of dead worlds by your own estimates, but it's all or nothing across universes. It will most probably just be safe.

If you had a definate number for both P(Doomsday Lottery Device Win) and P(Doomsday LHC) you'd shut up and multiply, but you haven't so you don't. But you still should because you're pretty sure P(D-LHC) >> P(DLDW) even if you don't know a figure for P(DLHC).

This assumes Paul's assumption, above.

Recently I did some probability calculations, starting with "made-up" numbers, and updating using Bayes' Rule, and the result was that something would likely happen which my gut said most firmly would absolutely not, never, ever, happen.

I told myself that my probability assignments must have been way off, or I must have made an error somewhere. After all, my gut couldn't possibly be so mistaken.

The thing happened, by the way.

This is one reason why I agree with RI, and disagree with Eliezer.

"The lottery would definately destroy worlds, with as many deaths as killing over six thousand people in each Everett branch."

We speak so casually about interpreting probabilities as frequencies across the many worlds, but I would suggest we need a rigorous treatment of what those other worlds are proportionally like before confidently doing so. (Cf. mine and Hal's commments in the June Open Thread.)

Unknown: I would REALLY like to know details.

Gunther Greindl: In my gut, I STRONGLY agree. My revealed preferences also match it. However, Philip Tetlocks' "Expert Political Judgment" tells me that among political experts, who have much better predictive powers than educated lay-people, specialists in X don't outperform specialists in Y in making predictions about X. This worries me A LOT. Another thing that worries me is that decomposing events exhaustively into their subcomponents makes the aggregate event seem more likely and it seems to me that by becoming an expert you come to automatically decompose events into their subcomponents.

Eliezer: I am pretty confident that it would be possible in principle, though not due to time constraints, to make a billion statements and get none wrong while keeping correlations fairly low.

I would not be comfortable with the inconsistency you describe about the lottery. I'm not sure how you can let it stand. I guess the problem is that you don't know which instinct to fix, and just reversing one belief at random is not going to improve accuracy on average.

Still, wouldn't careful introspection be likely to expose either some more fundamental set of inconsistent beliefs, that you can fix; or at least, to lead you to decide that one of the two beliefs is in fact stronger than the other, in which case you should reverse the weaker one? It seems unlikely that the two beliefs are exactly balanced in your degree of credence.

For the reactor, I'd say that the reasoning about one in a thousand odds is in fact a good way to go about analyzing the problem. It's how I approach other, similar issues. If I'm considering one of two routes through heavy traffic, I do roughly estimate the odds of running into a traffic jam. These are very crude estimates but they are better than nothing.

The biggest criticism I would give to such reasoning in this case is that as we go out the probability scale, we have much less experience, and our estimates are going to be far less accurate and calibrated. Furthermore, often in these situations we end up comparing or dividing probabilities, and error percentages go up astronomically in such calculations. So while the final figure may represent a mean, the deviation is so large that even slight differences in approach could have led to a dramatically different answer.

I would give substantially higher estimates that our theories are wrong - indeed by some measures, we know for sure our theories are wrong since they are inconsistent and none of the unifications work. However I'd give much lower estimates that the theories are wrong in just such a way that would lead to us destroying the earth.

I assume you were being facetious when you gave 75% odds that the authors would have maintained their opinion in different circumstances. Yet to me, it is a useful figure to read, and does offer insight into how strongly you believe. Without that number, I'd have guessed that you felt more strongly than that.

If P != NP and the universe has no source of exponential computing power, then there are evidential updates too difficult for even a superintelligence to compute - even though the probabilities would be quite well-defined, if we could afford to calculate them.


Trying to catch a flying ball, you're probably better off with your brain's built-in mechanisms, then [than?] using deliberative verbal reasoning to invent or manipulate probabilities.

There's more than just P != NP that defeats trying to catch a flying ball by predicting where it will land and going there. Or, for that matter, trying to go there by computing a series of muscular actions and then doing them. You can't sense where the ball is or what your body is doing accurately enough to plan, then execute actions with the precision required. A probability cloud perfectly calculated from all the available information isn't good enough, if it's bigger than your hand.

This is how to catch a ball: move so as to keep its apparent direction (both azimuth and elevation) constant.

But this doesn't mean you're going beyond probability theory or above probability theory.

It doesn't mean you're doing probability theory either, even when you reliably win. The rule "move so as to keep the apparent direction constant" says nothing about probabilities. If anyone wants to try at a probability-theoretic account of its effectiveness, I would be interested, but sceptical in advance.

There's more than just P != NP that defeats trying to catch a flying ball by predicting where it will land and going there. Or, for that matter, trying to go there by computing a series of muscular actions and then doing them.
You DO realize that some humans are perfectly capable of accomplishing precisely that action, right?

Eliezer, the correct way to resolve your inconsistency seems to be to be less approving of novel experiments, especially when they aren't yet necessary or probably very useful, and when a bit later we will likely have more expertise with regard to them. I refer to a comment I just made in another thread.

Can't give details, there would be a risk of revealing my identity.

I have come up with a hypothesis to explain the inconsistency. Eliezer's verbal estimate of how many similar claims he can make, while being wrong on average only once, is actually his best estimate of his subjective uncertainty. How he would act in relation to the lottery is his estimate influenced by the overconfidence bias. This is an interesting hypothesis because it would provide a measurement of his overconfidence. For example, which would he stop: The "Destroy the earth if God exists" lottery, or "Destroy the earth at odds of one in a trillion"? How about a quadrillion? A quintillion? A googleplex? One in Graham's number? At some point Eliezer will have to prefer to turn off the God lottery, and comparing this to something like one in a billion, his verbal estimate, would tell us exactly how overconfident he is.

Since the inconsistency would allow Eliezer to become a money-pump, Eliezer has to admit that some irrationality must be responsible for it. I assign at least a 1% chance to the possibility that the above hypothesis is true. Given even such a chance, and given Eliezer's work, he should come up with methods to test the hypothesis, and if it is confirmed, he should change his way of acting in order to conform with his actual best estimate of reality, rather than his overconfident estimate of reality.

Unfortunately, if the hypothesis is true, by that very fact, Eliezer is unlikely to take these steps. Determining why can be left as an exercise to the reader.

Unknown, describe the money pump. Also, are you the guy who converted to Christianity due to Pascal's Wager or am I thinking of someone else?

The tug-of-war in "How extreme a low probability to assign?" is driven, on the one hand, by the need for our probabilities to sum to 1 - so if you assign probabilities >> 10^-6 to unjustified statements of such complexity that more than a million of them could be produced, you will be inconsistent and Dutch-bookable. On the other hand, it's extremely hard to be right about anything a million times in a row.

My instinct is to look for a deontish human strategy for handling this class of problem, one that takes into account both human overconfidence and the desire-to-dismiss, and also the temptation for humans to make up silly things with huge consequences and claim "but you can't know I'm wrong".

Eliezer, you are thinking of Utilitarian (also begins with U, which may explain the confusion.) See http://utilitarian-essays.com/pascal.html

I'll get back to the other things later (including the money pump.) Unfortunately I will be busy for a while.

Was this speaker a believer in Discworldian probability theory? Which states, of course, that million-to-one chances come up 100% of the time, but thousand-to-one chances never. Maybe those numbers weren't plucked out of the air.

All we have to do is operate the LHC while standing on one foot, and the probability of the universe exploding will be nudged away from million-to-one (doesn't matter which direction - whoever heard of a 999,999-1 chance coming up?) and the universe will be saved.

Someone actually bought Pascal's wager? Oh boy. That essay looks to me like a perfect example of someone pulling oh-so-convenient numbers out of their fundament and then updating on them. See, it's math, I'm not delusional. *sigh*

Do you know what you get when you mix high energy colliders with Professor Otto Rossler?s charged micro black hole theory?

Answer: a golf ball (in 50 months to 50 years...)


Eliezer, the money pump results from circular preferences, which should exist according to your description of the inconsistency. Suppose we have a million statements, each of which you believe to be true with equal confidence, one of which is "The LHC will not destroy the earth."

Suppose I am about to pick a random statement from the list of a million, and I will destroy the earth if I happen to pick a false statement. By your own admission, you estimate that there is more than one false statement in the list. You will therefore prefer that I play a lottery with odds of 1 in a million, destroying the earth only if I win.

It makes no difference if I pick a number randomly between one and a million, and then play the lottery mentioned (ignoring the number picked.)

But now if I pick a number randomly between one and a million, and then play the lottery mentioned only if I didn't pick the number 500,000, while if I do pick the number 500,000, I destroy the earth only if the LHC would destroy the earth, then you would prefer this state of affairs, since you prefer "destroy the earth if the LHC would destroy the earth" to "destroy the earth with odds of one in a million."

But now I can also substitute the number 499,999 with some other statement that you hold with equal confidence, so that if I pick 499,999, instead of playing the lottery, I destroy the earth if this statement is false. You will also prefer this state of affairs for the same reason, since you hold this statement with equal confidence to "The LHC will not destroy the earth."

And so on. It follows that you prefer to go back to the original state of affairs, which constitutes circular preferences and implies a money pump.

I would advise, in most cases, against using non-numerical procedures to create what appear to be numerical probabilities. Numbers should come from numbers.

I very much disagree with this quote, and much of the rest of the post. Most of our reasoning about social stuff does not start from concrete numbers, so this rule would forbid my giving numbers to most of what I reason about. I say go ahead and pick a number out of the air, but then be very willing to revise it upon the slightest evidence that it doesn't fit will with your other numbers. It is anchoring that is the biggest problem. Being forced to pick numbers can be a great and powerful discipline to help you find and eliminate errors in your reasoning.

Unknown: God exists is not well specified. For something like "Zeus Exists" (not exactly that, some guy named Zeus does exist, and in some quantum branch there's probably an AGI that creates the world of Greek Myth in simulation) I would say that my confidence in its falsehood is greater than my confidence in the alleged probability of winning a lottery could be.

I say go ahead and pick a number out of the air,

A somewhat arbitrary starting number is also useful as a seed for a process of iterative approximation to a true value.

I strongly agree with Robin here. Thanks Robin for making the point so clearly.
I have to admit that not using numbers may be a better rule for a larger number of people than what they are currently using, as is majoritarianism, but neither is a good rule for people who are trying to reach the best available beliefs.

It is anchoring that is the biggest problem.

In the strongest form, that points directly opposite your advice.

I think that Robin was saying that anchoring, not the arbitrariness of starting points, is the big problem for transitioning from qualitative to quantitative thinking. You can make up numbers and so long as you update them well you get to the right place, but if you anchor too much against movement from your starting point but not against movement towards it you never get to an accurate destination.

I suspect my statement is the one that needed clarification.
I was measuring the size of a problem by the psychological difficulty of overcoming it. If anchoring is too big to overcome, it is better to avoid situations where it applies. And identifying the bias is not (necessarily) much of a step towards overcoming it.

Numbers are not needed for anchoring.
We could arrange the probabilities of the truth of statements into partially ordered sets.
This po set can even include statements about the probabilistic relation between statements.

Well, we should be careful to avoid the barbers paradox though...
things like x = {x is more likely then y} are a bad idea

I think it would be better to avoid just making up numbers until we absolutely have to, we actually find our selves playing a lottery for the continued existence of Earth, or there is some numerical process grounded in statistics that provides the numbers, resting on some assumptions. However, by anchoring probabilities in post sets we might get bounds on things for which we can not compute probabilities.

Me: There's more than just P != NP that defeats trying to catch a flying ball by predicting where it will land and going there. Or, for that matter, trying to go there by computing a series of muscular actions and then doing them.

Caledonian: You DO realize that some humans are perfectly capable of accomplishing precisely that action, right?

People can catch balls. Nobody can do it by the mechanism described. Fielders in ball games will turn away from the ball and sprint towards where they think it will come down, if they can't run fast enough while keeping it in sight, but they still have to look at the ball again to stand any chance of catching it. The initial sense data itself doesn't determine the answer, however well processed.

When what you need is a smaller probability cloud, calculating the same cloud more precisely doesn't help. Precision about your ignorance is not knowledge.

yes, numbers are not necessary for anchoring. I think that they make the anchoring worse, but it would be very bad to avoid numbers just because they make it easy to see anchoring.

I'll also disagree with the argument Eliezer gives here. See Robin's post. In addition to coming up with a probability with which we think an event will occur, we should also quantify how sure we are that that is the best possible estimate of the probability.

e.g. I can calculate the odds I'll win a lottery and if someone thinks their estimate of the odds is much better then (if we lack time or capital constraints) we can arrange bets about whose predictions will prove more accurate over many lotteries.

Re: Someone actually bought Pascal's wager? Oh boy.

E.g. see: Dinesh D'Souza, 8 minutes in.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30