## September 20, 2008

While I'm happy to have had the confidence of Richard, I thought my last comment could use a little improvement.

What we want to know is P(W|F,S)

As I pointed out F=> S so P(W|F,S) = P(W|F)

We can legitimately calculate P(W|F,S) in at least two ways:

1. P(W|F,S) = P(W|F) = P(F|W)P(W)/P(F) <- the easy way

2. P(W|F,S) = P(F|W,S)P(W|s)/P(F|S) <- harder, but still works

there are also ways you can get it wrong, such as:

3. P(W|F,S) != P(F|W,S)P(W)/P(F) <- what I said other people were doing last post

4. P(W|F,S) != P(F|W,S)P(W)/P(F|S) <- what other people are probably actually doing

In my first comment in this thread, I said it was a simple application of Bayes' rule (method 1) but then said that Eliezer's failure was not to apply the anthropic principle enough (ie I told him to update from method 4 to method 2). Sorry if anyone was confused by that or by subsequent posts where I did not make that clear.

Allan: your intuition is wrong here too. Notice that if Zeus were to have independently created a zillion people in a green room, it would change your estimate of the probability, despite being completely unrelated.

Eliezer: F => S -!-> P(X|F) = P(X|F,S)

All right, give me an example.

And yeah, anthropic reasoning is all about conditioning on survival, but you have to do it consistently. Conditioning on survival in some terms but not others = fail.

Richard: your first criticism has too low an effect on the probability to be significant. I was of course aware that humanity could be wiped out in other ways but incorrectly assumed that commenters here would be smart enough to understand that it was a justifiable simplification. The second is wrong: the probabilities without conditioning on S are "God's eye view" probabilities, and really are independent of selection effects.

Allan, oh ****, the elementary math in my previous comment is completely wrong. (In the scenario I gave, the probability that you have breast cancer is 1%, not 10%, before taking the test.) My argument doesn't even approximately work as given: if having breast cancer makes it more likely that you get a positive mammography, then *indeed* getting a positive mammography must make it more likely that you have breast cancer. Sorry!

(I'm still convinced that my argument re the LHC is correct, but I realize that I'm just looking stupid right now, so I'll just shut up for now :-))

Sorry Richard, well of course they aren't necessarily independent. I wasn't quite sure what you were criticising. But I pointed out already that, for example, a new physical law might in principle both cause the LHC to fail and cause it to destroy the world if it did not fail. But I pointed out that this was not what people were arguing, and assuming that such a relation is not the case then the failure of the LHC provides no information about the chance that a success would destroy the world. (And a small relation would lead to a small amount of information, etc.)

Oops, I fail! I thought F >= S meant "F is larger than S". But looking at the definitions of terms, Fail >= Survival must mean "Fail subset_of Survival". (I do protest that this is an odd symbol to use.)

Okay, looking back at the original argument, and going back to definitions...

If you've got two sets of universes side-by-side, one where the LHC destroys the world, and one where it doesn't, then indeed observing a long string of failures doesn't help tell you which universe you're in. However, after a while, nearly all the observers will be concentrated into the non-dangerous universe. In other words, if you're going to start running the LHC, then, conditioning on your own survival, you are nearly certain to be in the non-dangerous universe. Then further conditioning on the long string of failures, you are equally likely to be in either universe. If you start out by conditioning on the long string of failures, then conditioning on your own survival indeed doesn't tell you anything more.

But under anthropic reasoning, the argument doesn't play out like this; the way anthropic reasoning works, particularly under the Quantum Suicide or Quantum Immortality versions, is something along the lines of, "You are never surprised by your own survival".

From the above, we can see that we need something like:

Initial probability of Danger: 50%
Initial probability of subjective Survival: 100%
Probability of Failure given Danger and Survival: 100%
Probability of Failure given ~Danger and Survival: 1%
Probability of Danger given Survival and Failure: ~1%

So to comment through Simon's logic vs. anthropic logic step by step:

First thing to note is that since F => S, we have P(W|F) = P(W|F,S), so we can just work out P(W|F)

still holds technically true

Bayes:

P(W|F) = P(F|W)P(W)/P(F)

Still technically true; but once you condition on survival, as anthropics does in effect require, then P(Fail|Danger) is very high.

Note that none of these probabilities are conditional on survival. So unless in the absence of any selection effects the probability of failure still depends on whether the LHC would destroy Earth, P(F|W) = P(F), and thus P(W|F) = P(W).

Here we depart from anthropic reasoning. As you might expect, quantum suicide says that P(Fail|Danger) != P(Fail). That's the whole point of raising the possibility of, "given that the LHC might destroy the world, how unusual that it seems to have failed 50 times in a row"

In effect what Eliezer and many commenters are doing is substituting P(F|W,S) for P(F|W). These probabilities are not the same and so this substitution is illegitimate.

...but as stated originally, conditioning on the existence of "observers" is what anthropics is all about. It's not that we're substituting, but just that all our calculations were conditioned on survival in the first place.

Eliezer, I used "=>" (intending logical implication), not ">=".

I would suggest you read my post above on this second page, and see if that changes your mind.

Also, in a previous post in this thread I argued that one should be surprised by externally improbable survival, at least in the sense that it should make one increase the probability assigned to alternative explanations of the world that do not make survival so unlikely.

Eliezer, I used "=>" (intending logical implication), not ">=".

Zis would seems to explains it.

(I use -> to indicate logical implication and => to indicate a step in a proof, or otherwise implication outside the formal system - I do understand this to be conventional.)

I would suggest you read my post above on this second page, and see if that changes your mind.

Not particularly. I use 4 but with P(W|S) = P(W) which renders it valid. (We're not talking about two side-by-side universes, but about prior probabilities on physical law plus a presumption of survival.)

Also, in a previous post in this thread I argued that one should be surprised by externally improbable survival, at least in the sense that it should make one increase the probability assigned to alternative explanations of the world that do not make survival so unlikely.

This could only reflect uncertainty that anthropic reasoning was valid. If you were certain anthropic reasoning were valid (I'm sure not!) then you would make no such update. In practice, after surviving a few hundred rounds of quantum suicide, would further survivals really seem to call for alternative explanations?

After surviving a few hundred rounds of quantum suicide the next round will probably kill you.

Are you familiar with the story of the man who got the winning horse race picks in the mail the day before the race was run? Six times in a row his mysterious benefactor was right, even correctly calling a victory for a horse with forty-to-one odds. Now he gets an envelope in the mail from the same mysterious benefactor asking for \$1,000 in exchange for the next week's picks. Are you saying he should take the deal and clean up?

Not particularly. I use 4 but with P(W|S) = P(W) which renders it valid. (We're not talking about two side-by-side universes, but about prior probabilities on physical law plus a presumption of survival.)

You mean you use method 2. Except you don't, or you would come to the same conclusion that I do. Are you claiming that P(W|S)= P(W)? Ok, I suspect you may be applying Nick Bostrom's version of observer selection: hold the probability of each possible version of the universe fixed independent of the number of observers, then divide that probability equally amongst the observers. Well, that approach is BS whenever the number of observers differs between possible universes, since if you imagine aliens in the universe but causally separate, the probabilities would depend on their existence.

Also, does it really make sense to you, intuitively, that you should get a different result given two actually existing universes compared to two possible universes?

This could only reflect uncertainty that anthropic reasoning was valid. If you were certain anthropic reasoning were valid (I'm sure not!) then you would make no such update. In practice, after surviving a few hundred rounds of quantum suicide, would further survivals really seem to call for alternative explanations?

As I pointed out earlier, if there was even a tiny chance of the machine being broken in such a way as to appear to be working, that probability would dominate sooner or later.

One last thing: if you really believe that annihilational events are irrelevant, please do not produce any GAIs until you come to your senses.

Whoops, I didn't notice that you did specifically claim that P(W|S)=P(W).

Do you arrive at this incorrect claim via Bostrom's approach, or another one?

This is a subject I've long been meaning to give some thought too, but at the moment I'm pretty swamped - hope to get back to it when I have more time.

Simon, pretty much Bostrom's approach. Self-Sampling without Self-Indication. I know it's wrong but I don't have any better approach to take.

Why do you reject self-indication? As far as I can recall the only argument Bostrom gave against it was that he found it unintuitive that universes with many observers should be more likely, with absolutely no justification as to why one would expect that intuition to reflect reality. That's a very poor argument considering the severe problems you get without it.

I suppose you might be worried about universes with many unmangled worlds being made more likely, but I don't see what makes that bullet so hard to bite either.

Wasn't one of the conclusions we arrived at in the quantum mechanics sequence that "observer" was a nonsense, mystical word?

I might add, for the benefit of others, that self-sampling forbids playing favourites among which observers to believe that you are in a single universe (beyond what is actually justified by the evidence available), and self-indication forbids the same across possible universes.

Nominull: It's a bad habit of some people to say that reality depends on, or is relative to observers in some way. But even though observers are not a special part of reality, we are observers and the data about the universe that we have is the experience of observers, not an outside view of the universe. So long as each universe has no more than one observer with your experience, you can take your experience as objective evidence that you live in a universe with one such observer instead of zero (and with this evidence to work with, you don't need to talk about observers). But it's difficult to avoid talking about observers when a universe might have multiple observers with the same subjective experience.

in a previous [comment] in this thread I argued that one should be surprised by externally improbable survival, at least in the sense that it should make one increase the probability assigned to alternative explanations of the world that do not make survival so unlikely.

Simon, I think that the previous comment you refer to was the smartest thing anyone has said in this comment section. Instead of continuing to point out the things you got right, I hope you do not mind if I point out something you got wrong, namely,

Richard: your first criticism has too low an effect on the probability to be significant. I was of course aware that humanity could be wiped out in other ways but incorrectly assumed that commenters here would be smart enough to understand that it was a justifiable simplification.

It is not a justifiable simplification. A satisfactory answer to the question you were trying to answer should remain satisfactory even if other existential risks (e.g., a giant comet) are high. If other existential risks were high, would you just throw up your hands and say that the question you were trying to answer is unanswerable?

Again, I think your contributions to this comment thread were better than anyone else's. I hope you continue to contribute here.

Allan: your intuition is wrong here too. Notice that if Zeus were to have independently created a zillion people in a green room, it would change your estimate of the probability, despite being completely unrelated.

I don't see how, unless you're told you could also be one of those people.

Benja: Allan, you are right that if the LHC would destroy the world, and you're a surviving observer, you will find yourself in a branch where LHC has failed, and that if the LHC would not destroy the world and you're a surviving observer, this is much less likely. But contrary to mostly everybody's naive intuition, it doesn't follow that if you're a surviving observer, LHC has probably failed.

I don't believe that's what I've been saying; the question is whether the LHC failing is evidence for the LHC being dangerous, not whether surviving is evidence for the LHC having failed.

Richard, obviously if F does not imply S due to other dangers, then one must use method 2:

P(W|F,S) = P(F|W,S)P(W|S)/P(F|S)

Let's do the math.

A comet is going to annihilate us with a probability of (1-x) (outside view) if the LHC would not destroy the Earth, but if the LHC would destroy the Earth, the probability is (1-y) (I put this change in so that it would actually have an effect on the final probability)
The LHC has an outside-view probability of failure of z, whether or not W is true
The universe has a prior probabilty w of being such that the LHC if it does not fail will annihilate us.

Then:
P(F|W,S) = 1
P(F|S) = (ywz+x(1-w)z)/(ywz+x(1-w)z+x(1-w)(1-z))
P(W|S) = (ywz)/(ywz+x(1-w)+x(1-w)(1-z))

so, P(W|F,S) = ywz/(ywz+x(1-w)z) = yw(yw+x(1-w))

I leave it as an exercise to the reader to show that there is no change in P(W|F,S) if the chance of the comet hitting depends on whether or not the LHC fails (only the relative probability of outcomes given failure matters).

Really though Richard, you should not have assumed in the first place that I was not capable of doing the math. In the future, don't expect me to bother with a demonstration.

Allan: you're right, I should have thought that through more carefully. It doesn't make your interpretation correct though...

I have really already spent much more time here today than I should have...

Err... I actually did the math a silly way, by writing out a table of elementary outcomes... not that that's silly itself, but it's silly to get input from the table to apply to Bayes' theorem instead of just reading off the answer. Not that it's incorrect of course.

And by elementary I mean the 8 different ways W, F, and the comet hit/non hit can turn out.

Allan: I don't believe that's what I've been saying; the question is whether the LHC failing is evidence for the LHC being dangerous, not whether surviving is evidence for the LHC having failed.

I was trying to restate in different terms the following argument for failure to be considered evidence:

The intuition on my side is that, if you consider yourself a random observer, it's amazing that you should find yourself in one of the extremely few worlds where the LHC keeps failing, unless the LHC is dangerous, in which case all observers are in such a world.

For "observer" I substituted "surviving observer," because when doing the math I find it more helpful to consider all potential observers and then say that some of them are dead and thus can't observe anything. So my "surviving observer" is the same as your "observer," right?

So I read your argument as: If the LHC is benign, and you're a random (surviving) observer, then it's amazing if (i.e., there is a low probability that) you find yourself in one of the few worlds where the LHC keeps failing. If the LHC is dangerous, and you're a random observer, then it's non-amazing (i.e., there is a high probability that) you find yourself in a world where the LHC keeps failing. Therefore, if you're a random observer, and you find yourself in a world where the LHC keeps failing, then the LHC is probably dangerous (because then, we don't need to assume something amazing going on). Am I misunderstanding something?

If I understand you right, what I'm saying is that both the if's are clearly correct, but I believe that the 'therefore' doesn't follow.

To me, the problem is essentially the same as the following: You are one of 10,000 people who have been taken to a prison. Nobody has explained why. Every morning, the guards randomly select 9/10 of the remaining prisoners and take them away, without explanation. Among the prisoners, there are two theories: one faction thinks that the people taken away are set free. The other faction thinks that they are getting executed.

It is the fourth morning. You're still in prison. The nine other people who remained have just been taken away. Now, if the other people have been executed, then you are the only remaining observer, so if you're a random observer, it's not surprising that you should find yourself in prison. But if the other people have been set free, then they're still alive, so if you're a random observer, there is only a 1/10,000 chance that you are still in prison. Both of these statements are correct if you are a random (surviving) observer. But it doesn't follow that you should conclude that the other people are getting shot, does it? (Clearly you learned nothing about that, because whether or not they get shot does not affect anything you're able to observe.)

Now, I get that you probably think something makes this line of reasoning not apply when we consider the anthropic principle (although I do think that you're wrong then :)). But my point is that, unless I'm missing something, the probabilistic reasoning is the same as in my restatement of your argument, so if the laws of probability don't make the conclusion follow in this scenario, they don't make the conclusion follow in your argument, either.

I should say that I don't reject "the" anthropic principle. I wholeheartedly embrace the version of it that I can derive from the kind of reasoning as above. For example: If our theory of evolution seems to suggest that there is one very improbable step in the evolution of intelligent life -- so improbable that it's not likely to have happened even a single time in the history of the universe -- should we then take that as a reason to conclude that something is wrong with our theory? If we are pretty sure that there is only a single universe, yes. If we have independent evidence that all possible Everett branches exist, no. (If something like mangled worlds is true, maybe -- but let's not get into that now...)

Why should we reject our theory in a single universe, but not if all Everett branches exist? Consider again the prison analogy. You observed how the guards chose the prisoners to take away, and it sure looked random. But now you are the only surviving prisoner. Should you conclude that the guards' selection process wasn't really random? There's no reason to: If the guards used a random process, one prisoner had to remain on the fourth day, and this may just as well have been you -- nothing surprising going on. This corresponds to the scenario where all possible Everett branches exist.

But suppose that you were the only prisoner to begin with (and you know this), and every morning the guards threw a ten-sided die which is marked "keep in prison" on one side and "take away" on the nine others -- and it came up "keep in prison" every morning. In this case, it seems to me that you do have a reason to start suspecting that the die is fixed (i.e., that your original theory, that the "keep in prison" outcome had only a 10% chance of happening, was wrong). This corresponds to the scenario where there is only a single universe.

This is how I always understood the anthropic principle when reading about it, and this version of it I embrace. The other version I'm pretty sure is wrong.

That said, if you have the energy to do so, please do keep arguing with me! :-) I don't really understand this "other anthropic principle," and I'm rejecting it simply because it disagrees with my calculations and I'm really pretty sure that I'm applying my probability theory right here. If I'm wrong, that will be humbling, but I would still rather know than not know, please :-)

My prior probability for the existence of a secret and powerful crackpot group willing to sabotage the LHC to prevent it from "destroying the world" is larger than my prior probabilty for the LHC-actually-destroying-the-world scenarios being true

Alejandro has a good point.

Benja: But it doesn't follow that you should conclude that the other people are getting shot, does it?

I'm honestly not sure. It's not obvious to me that you shouldn't draw this conclusion if you already believe in MWI.

(Clearly you learned nothing about that, because whether or not they get shot does not affect anything you're able to observe.)

It seems like it does. If people are getting shot then you're not able to observe any decision by the guards that results in you getting taken away. (Or at least, you don't get to observe it for long - I'm don't think the slight time lag matters much to the argument.)

I did a calculation here:
http://tinyurl.com/3rgjrl
and concluded that I would start to believe there was something to the universe-destroying scenario after about 30 clear, uncorrelated mishaps (even when taking a certain probability of foul play into account).

...Allan, sorry for the delay in replying. Hopefully tomorrow. (In my defense, I've spent the whole day seriously thinking about the problem ;-))

OK, I've finally had a little time to go over these comments and I am now persuaded to take the position of simon and Benja Fallenstein. I'd already decided to be a Presumptuous Philosopher and accept self-indication, and this just supports that further.

To me, the problem is essentially the same as the following: You are one of 10,000 people who have been taken to a prison. Nobody has explained why. Every morning, the guards randomly select 9/10 of the remaining prisoners and take them away, without explanation. Among the prisoners, there are two theories: one faction thinks that the people taken away are set free. The other faction thinks that they are getting executed.

It is the fourth morning. You're still in prison. The nine other people who remained have just been taken away. Now, if the other people have been executed, then you are the only remaining observer, so if you're a random observer, it's not surprising that you should find yourself in prison. But if the other people have been set free, then they're still alive, so if you're a random observer, there is only a 1/10,000 chance that you are still in prison. Both of these statements are correct if you are a random (surviving) observer. But it doesn't follow that you should conclude that the other people are getting shot, does it? (Clearly you learned nothing about that, because whether or not they get shot does not affect anything you're able to observe.)

An excellently clear way of putting it!

*bites bullet*

I suspect that anthropics is easy to solve if you think in terms of cognitive decision theory.

Okay, after reading several of Nick Bostrom's papers and mulling about the problem for a while, I think I may have sorted out my position enough to say something interesting about it. But now I'm finding myself suffering from a case of writer's block in explaining it, so I'll try to pull a small-scale Eliezer and say it in a couple of hiccups, rather than one fell swoop :-)

I have been significantly wrong at least twice in this thread, the first time when I thought everybody was reasoning from the same definitions as me, but getting their math wrong, and the second time when I said I held my view because I was "pretty sure I [was] applying my probability theory right". I had an intuition and a formal argument, but then I found that the two disagree in some edge cases, and I decided to retain the intuition, so my formal argument was not the solid rock I thought it was. All of which is a long-winded way of saying, it's about time that I concede that I may still be wrong about this, and if so, please do help me figure it out...

We all seem to agree that the issue depends on whether we accept self-indication, and that self-indication is equivalent to being a thirder in the Sleeping Beauty problem. When I first learned about this problem from Robin's post, I was very convinced that the halfer view was right -- to the tune of having been willing to bet money on it -- for about fifteen minutes. Then I thought about something like the following variation of it:

Beauty is put to sleep on Sunday, and a fair coin is tossed. Beauty is awakened twice, once on Monday and once on Tuesday; in between, she is given an amnesia-inducing drug, so that when she wakes up, she cannot tell whether she has been woken up before. One minute after Beauty wakes up, a light flashes. If it is Tuesday, and the coin came up heads, the light is red; otherwise, it is blue.

When Beauty wakes up, before the light flashes, what is her subjective probability that (h1) the coin came up heads, and it's Monday; (h2) heads, Tuesday; (t1) tails, Monday; (t2) tails, Tuesday?

I cannot conceive of a reason not to assign the probability 1/4 to each of these propositions, and in my opinion, when Beauty sees the light flash red, she must update her subjective probability in the obvious way (or the notion of subjective probability no longer makes much sense to me). Then, of course, after seeing the light flash blue, Beauty's probability that the coin fell heads is 1/3.

Short of assigning special ontological status to being consciously awake, I don't see a way to distinguish between the original Sleeping Beauty and my variation after the light flashes blue, so I'm a thirder now. My new view is that observing the random variable (color=blue) can change my probability in non-mysterious ways, so observing the random variable (awake=yes) can, too.

In his paper on the problem, Nick argues for a "solution" that would apply to my version, too. He would reject my view of how Beauty must update her probabilities if she sees a blue light. His argument goes something like this:

What I really need to consider is all of Beauty's observer-moments in all possible worlds; Beauty has a prior over these moments, considers the evidence she has for which moment she is in, and does a Bayesian update. The moment when Beauty wakes up is different from the moment when the light flashes, so she needs to consider at least eight possible moments: (h1-) heads, Monday, she wakes up; (h1+) heads, Monday, the light flashes; and so on. Nothing in the axioms of probability theory requires the probability of (h1+) to be related in any way to the probability of (h1-)! In fact, Nick would argue, we should simply assign probabilities like this:

p(xx- | h1- \/ h2- \/ t1- \/ t2-) = 1/4 (for xx in {h1,h2,t1,t2})
p(h1+ | h1+ \/ t1+ \/ t2+) = 1/2
p(xx+ | h1+ \/ t1+ \/ t2+) = 1/4 (for xx in {t1,t2})

I agree that this is formally consistent with the axioms of probability, but in order for Beauty to be rational, in my opinion she must still update her probability estimate in the "normal" way when the light flashes blue. Nick's approach strikes me as saying, "I'm a completely new observer-moment now, why should I care about my probability estimates a minute ago?" If our formalism allows us to do that, I think our formalism isn't strong enough. In this case, I'd require that

p(xx- | h1- \/ h2- \/ t1- \/ t2-)
= p(xx+ | h1+ \/ h2+ \/ t1+ \/ t2+)

--i.e., before conditioning on the actual colors she sees, Beauty's probability estimates when the light flashes must be the same as when she wakes up. I don't know how well this generalizes, but if we accept it in this case, it blocks Nick's proposal.

Anybody here who finds Nick's solution intuitively right?

It may be silly to continue this here, since I'm not sure anybody's still reading, but at least I'm writing it down at all this way, so... here's "Nick's Sleeping Beauty can be Dutch Booked" (by Nick's own rules)

In his Sleeping Beauty paper, Nick considers the ordinary version of the problem: Beauty is awakened on Monday. An hour later, she is told that it is Monday. Then she is given an amnesia drug and put to sleep. A coin is flipped. If the coin comes up tails, she is awakened again on Tuesday (and can't tell the difference to Monday). Otherwise, she sleeps through to Wednesday.

Nick distinguishes five possible observer-moments: Beauty wakes up on Monday (h1 and t1, depending on heads/tails); Beauty is told that it's Monday (h1m and t1m); Beauty wakes up on Tuesday (t2). Let P-(x) := P(x | h1 \/ t1 \/ t2), and P+(x) := P(x | h1m \/ t1m).

There are two possible worlds, heads-world (h1,h1m) and tails-world (t1,t1m,t2). Within each of the groups (h1,t1,t2) and (h1m,t1m), Nick assigns equal probabilities to each observer-moment in a given possible world. This gives:

P-(h1) = 1/2; P-(t1) = 1/4; P-(t2) = 1/4
P+(h1m) = 1/2; P+(t1m) = 1/2

In his paper, Nick considers the following Dutch book, suggested by a referee (I'm quoting from the paper):

Upon awakening, on both Monday and Tuesday, before either knows what day it is, the bookie offers Beauty the following bet: Beauty gets \$10 if HEADS and MONDAY. Beauty pays \$20 if TAILS and MONDAY. (If TUESDAY, then no money changes hands.) On Monday, after both the bookie and Beauty have been informed that it is Monday, the bookie offers Beauty a further bet: Beauty gets \$15 if TAILS. Beauty pays \$15 if HEADS. If Beauty accepts these bets, she will emerge \$5 poorer.

Nick dismisses this argument because if the coin falls tails, Beauty will accept the first bet twice, once on Monday and once on Tuesday. Now, on Tuesday no money changes hands, so what's the difference? Well, Nick thinks it's very interesting that it could make a difference, but clearly it does, you see, because otherwise Sleeping Beauty could be Dutch booked if she accepts his probability assignments!

Instead of trying to argue that it makes no difference, let me just exhibit a variation where Beauty only accepts every bet at most once in every possible world.

Before Beauty is put to sleep, we throw a second fair coin, labelled A and B. If it comes up A, then on Monday, we tell Beauty, "It's day A!" And if we wake her up on Tuesday, we tell her, "It's day B!" If the coin comes up B, Monday is B, and Tuesday is A.

We now have doubled the number of worlds and observer-moments. The worlds are HA, HB, TA, and TB, each with probability 1/4; the observer-moments are ha1, ha1m; hb1, hb1m; ta1, ta1m, ta2; tb1, tb1m, tb2. P- and P+ are defined analogously to before, and again, we assign equal probability to each of the awakenings in every possible world (and make them sum to the probability of that world). This gives:

P-(ha1) = P-(hb1) = 1/4
P-(ta1) = P-(ta2) = P-(tb1) = P-(tb2) = 1/8
P+(ha1m) = P+(hb1m) = P+(ta1m) = P+(tb1m) = 1/4

The sets of observer-moments that Beauty cannot distinguish are: {ha1,ta1,tb2}; {hb1,tb1,ta2}; {ha1m,ta1m}; {hb1m,tb1m}. (E.g., on {ha1,ta1,tb2}, Beauty just knows that she's been awakened and that it's "Day A." In world B, Tuesday is Day A, thus tb2 is in this set.)

Note well that in none of these sets, there is more than one observer-moment from the same possible world. I exhibit the following variation of the above Dutch Book.

On {ha1,ta1,tb2}, the Bookie offers Beauty the first bet above: Beauty gets \$10 if HEADS and MONDAY. Beauty pays \$20 if TAILS and MONDAY. (If TUESDAY, then no money changes hands.)

On {ha1m,ta1m}, the Bookie offers the second bet: Beauty gets \$15 if TAILS. Beauty pays \$15 if HEADS.

Beauty now loses \$5 if the day-label-coin comes up A, and breaks even if it comes up B. Every bet is accepted exactly once in every possible world in which it is offered at all. We could add symmetrical additional bets to make sure that Beauty also loses money in B worlds, but I think I've made my point. Nick can create his priors over observer-moments without violating the axioms of probability, but if it worries him if Beauty can be Dutch-booked in the way he discusses in his paper, I do believe he needs to be worried...

So if I think that (something like) the Self-Indication Assumption is correct, what about Nick's standard thought experiment in which the silly philosopher thinks she can derive the size of the cosmos from the fact she's alive?

Well, the experiment does worry me, but I'd like to note that self-sampling without self-indication produces, in fact, a very similar result (if the reference class is all conscious observers, which Nick's version of the experiment seem to assume). I give you The Presumptuous Philosopher and the Case of the Twin Stars:

Physicists have narrowed down the search for the Theory of Everything to T1 and T2, between which considerations of super-duper-symmetry are indifferent. We know that the cosmos is very big, to the tune of containing a trillion trillion galaxies, most of which are expected to contain life. But there's a twist: According to T1, all but one in a trillion galaxies should consist entirely of twin star systems. T2 does not make this prediction. Physicists are preparing to do a simple test that would decide between the theories. Enter the Presumptuous Philosopher: "Guys, our galaxy has lots of single-star systems. The conditional probability of this if T1 is true and we're a random sample from all conscious observers is only one in a trillion! Stop doing this silly experiment and do something else instead!"

If you accept this thought experiment (which requires only self-sampling) but reject a variation where T1 is ruled out because it predicts that cosmological death rays will make life impossible in all galaxies but one in a trillion (which requires self-sampling), then I think you've allowed yourself to be suckered into implicitly assuming that conscious observation is something ontologically fundamental. Though I accept that you may not be convinced of this yet :-)

(Side note: Lest you be biased against the philosopher just because she dares to apply probability theory, do also consider the case where T1 predicts that Mars had a chance of 4/5 per year of flying out of the solar system since it came into existence -- and beat those odds by random chance every single time. Of course, in that case, the physicists would already be convinced that her reasoning is sound, to the tune that they would already have applied it itself.)

In my previous comment, I mentioned my worry that accepting observer self-sampling without self-indication means that you've been suckered into taking conscious observation as an ontological primitive. (Also, I've been careful not to use examples that involve the size of the cosmos.) I would like to suggest that instead of a prior over observer-moments in possible worlds, we start with a prior over space-time-Everett locations in possible worlds. If all possible worlds we consider have the same set of space-time-Everett locations, and we have a prior P0 over possible worlds, then I suggest that we adopt the prior over (world, location) pairs:

P((w,x)) = P0(w) / number of possible locations

(Actually, that's not necessarily quite right: If the "amplitude as degree of reality" interpretation is true, Everett branches should of course be weighted in the obvious way.)

As with observer-moments, we then condition on all the evidence we have about our actual space-time-Everett location in our actual possible world, and call the result our "subjective probability" distribution.

Isn't anthropic reasoning about taking into account the observer selection effects related to the fact that we are conscious observers? Sure, but it seems to me that any non-mysterious anthropic reasoning is taken care of just fine by the conditioning step. Any possible worlds, Everett branches and cosmic regions that don't support intelligent life will automatically be ruled out, for example.

The above definition trivially implies the following weak principle of self-indication:

If all possible worlds we consider have the same set of locations, worlds that contain more locations consistent with our evidence will tend to be more likely after conditionalization. (To be precise, the probability of each world w is weighted by P0(w) * number of locations in w consistent with our evidence).

This principle is enough to support being a thirder in the Sleeping Beauty problem, for example (which was what originally suggested it to me, when I was wondering what prior Beauty should update when she observes herself to be awake).

So what if we are uncertain about the size of the universe (so that its size depends on which possible world we are in)? Then we are faced with the same question as before: Should we treat finding ourselves in bigger universes as more probable a priori, or not?

Formally, the question we face is, if we have a prior P0 over possible worlds, what should our prior over (possible world, space-time-Everett location) pairs be?

Physical self-sampling without self-indication. P((w,x)) = P0(w) / number of possible locations in world w

Physical self-sampling with physical self-indication. P((w,x)) = alpha * P0(w), where alpha is a normalization constant (alpha = 1 / Sum_w'. P0(w') * number of possible locations in world w')

(As before, we may want to weigh Everett branches in the obvious way.) Both of these definitions give us the weak principle of self-indication (defined in the previous comment), since they agree with the previous comment's definition when all possible worlds contain the same number of locations. So they both support thirding in Sleeping Beauty.

But which of the definitions should we adopt? Note that sampling without self-indication has the property that P(w) = P0(w), i.e., before we condition on any evidence (including the fact that we are conscious observers), the probability of finding ourselves in world w is exactly the probability of that world, according to P0. On the face of it, this sounds exactly like what we mean by having a prior P0 over the possible worlds.

I think we may mean different things with P0 depending on how we arrive at P0, though. But for the moment, let me note that while the principle of weak self-indication forces me to accept the presumptuous philosopher's position in both the Case of the Twin Stars and the Case of the Death Rays, I may still have a good reason to reject the conclusion that the cosmos is infinite with probability one.

Unfortunately, physical self-sampling without self-indication has odd consequences of its own. Consider the following thought experiment:

Physicists have conclusively figured out what the theory of everything is. We know roughly how the cosmos will behave until a trillion years into the future. However, it's still unclear what will happen at this point: either (T1) the universe will end, or (T2) the universe will continue for another trillion trillion years, but be unable to support intelligent life. A hard mathematical calculation can show which of these is true, but before doing the calculation, each theory has a 1/2 prior probability (in the same sense that before doing the calculation, you have a 1/10 subjective probability that the trillionth decimal digit of pi is a seven).

Physicists want to schedule supercomputer time to determine the answer. Enter Presumptuous: "By physical self-sampling, the probability of T2 given our observations is only about one in a trillion. This calculation is a waste of money!"

She calculates as follows. P0(T1) = P0(T2) = 1/2. According to T2, the universe contains a trillion more space-time locations than according to T1. But according to both theories, the universe contains only one location consistent with our evidence. According to the definition given in the previous comment, this makes T2 much less likely that T1.

Intuitively, the argument is, "According to T2, there are a trillion more places we could have found ourselves at (at most of which we would not have been conscious observers, but taking that into account would be supernatural wonder tissue). So having found ourselves at this particular place is much more surprising according to T2."

But this argument doesn't sound very convincing to me. From where do we get this supposed lottery over space-time locations? At least, the argument sounds much less intuitively convincing than the following: "Our uncertainty is mathematical, and our observations would be exactly the same according to each theory -- we can't conclude anything about the mathematical result from the fact that one would destroy the universe, while the other would only leave it barren."

In the next comment, I'll develop that intuition into a more formal argument supporting self-indication.

As you know, I don't spend much time worrying about the Large Hadron Collider when I've got much larger existential-risk-fish to fry.
----
After observing empirically that the LHC had failed 100 times in a row, would you endorse a policy of keeping the LHC powered up, but trying to fire it again only in the event of, say, nuclear terrorism or a global economic crash?
----

The real question, Eliezer, is how many times the LHC would have to fail before you decide to fundamentally change the direction of your research? At some point the most profitable avenue of research in the pursuit of friendly AI would become the logistics of combining a mechanism for quantum suicide with a random number generator. Would you shut up, multiply and then invest the entirety of your research on the nuances of creating a secure, hostile-AI-preventing universe-suicide bunker? Let a RNG write the AI and save-scum yourself to friendly AI paradise!

Pardon me, my question skipped far too many inferential steps for me to be comfortable that my meaning is clear. Allow me to query for the underlying premises more clearly:

* Is quantum-destroying-the-entire-universe suicide different to plain quantum-I-killed-myself-in-a-box suicide?

That is to say, does Eliezer consider it rational to optimise for the absolute tally of Everett branches or the percentage of them? In "The Bottom Line" Elizier gives an example definition of my effectiveness as a rationalist as how well my decision optimizes the percentage of Everett branches that don't get me killed by faulty brakes.

Absurd as it may be, let us say that the LHC, or perhaps the LHSM (Large Hadron Spaghetti Monster) destroys the entire universe. If a particular Everett branch is completely obliterated by the Large Hadron Spaghetti Monster then do I still count those branches when I'm doing my percentage or do I only count the worlds where there is an actual universe there to count?

I can certainly imagine some consider parts of the Everett tree that are not in existence in any manner at all to have been 'pruned', think the end result is kind of neat and so decide that their utility function optimizes according to the number of Everett branches that contain the desired outcome divided by the number of Everett branches among those that exist. Another could say that they simply want to maximise the absolute number of Everett branches that contain desired outcomes. The LHSM eating an entire Everett branch would be the same as any other particular event making an Everett branch undesirable.

The above is my intuitive interpretation of the core of Elizier's parting question. Obviously, if we are optimizing for percentage of Everett branches then it is rational to create an LHSM rigged to eat the branch if it contains nuclear terrorism, global economic crashes or Elizier accidentally unleashing the Replicators upon us all. If, however, we are optimizing by absolute desirable Everett branch count then rigging the LHSM to fire whenever the undesirable outcome occurs is merely a waste of resources.

* Are there fates worse than the universe being obliterated?

I note this question simply to acknowledge that other factors could weigh in to the answer to Elizier's question than the significant one of whether we count not-actually-Everett branches. Perhaps Joe considers the obliteration of the universe to be an event like all others. The analogy would perhaps be to having x% of Everett branches go off in straight line together all rather transparent and not going any place in particular while the remaining (100-x)% of Everett branches head off in the typical way. Joe happens to assign -100 utility to the universe being obliterated, +2 to getting a foot massage and -3,000 to being beaten by a girl. Joe would multiply the x% of the not-Everett branch by -100 and (1-x)% by -3000. It would be rational for Joe to create a LHSM that would be fired whenever he suffered the feared humiliation. That is, unless he anticipated 1,450 foot massages in return for keeping the universe intact!

It seems to me that in order for it to be rational to LHSM the universe on the event of nuclear terrorism or global economic collapse and yet not rational to use the LHSM to make a friendly AI then:

* Universe obliteration must be evaluated as a standard Everett branch.
AND
* Universe obliteration must be assigned a greater utility than nuclear terrorism or global economic collapse.
(This possibility is equivalent to standard quantum suicide with a caveat that tails was going to give you cancer anyway and you're rather be dead.)

OR, ALTERNATIVELY

* Universe obliteration is different from quantum suicide. Obliterated universes don't count at all in the utility function so preventing nuclear terrorism by obliterating the universe makes the average world a better place once you do the math.
AND
* The complications involved in using the same LHSM to create a friendly AI are just not worth the hassle. (Or otherwise irrational for some reason that is unrelated to the rather large amount of universe obliteration that would be going on.)

Eliezer never implied an answer on whether he would fire a universe destroying LHC to prevent disaster. I wonder, if he did endorse that policy, would he also endorse using the same mechanism to further his research aim?

At some point the most profitable avenue of research in the pursuit of friendly AI would become the logistics of combining a mechanism for quantum suicide with a random number generator.

Usually learning new true information increases a person's fitness, but learning about the many-worlds interpretation seems to decrease the fitness of many who learn it.

Am I to assume then, Richard, that you consider the destruction of a branch entirely using whatever mechanism the LHC was supposedly going to destroy the fabric of reality is exactly equivalent to a more mundane death in a box? Or, did you simply use your cached thought regarding quantum suicide and saw a chance to be rude? I've got a hunch that it's the latter since the implication doesn't logically follow.

Dull, I was hoping something more useful to tell me. The implications of whatever the LHC could supposedly do and in particular why ever someone would choose blowing up the universe in preference to a couple of nukes going off were intriguing.

Incidentally, whatever makes you claim that 'new information increases a person's fitness'? Education notoriously reduces the rate of breeding in humans. I also haven't found many people who actually apply their information about evolution and make it their full time occupation to find places through which they can donate sperm.

OK, my previous comment was too rude. I won't do it again, OK?

Rather than answer your question about fitness, let me take back what I said and start over. I think you and I have different terminal values.

I am going to assume -- and please correct me if I am wrong -- that you assign an Everett branch in which you painless wink out of existence a value of zero (neither desirable or undesirable) and that consequently, under certain circumstances (e.g., at least one alternative Everett branch remains in which you survive) you would prefer painlessly winking out of existence to enduring pain.

My objection to this talk of destroying the universe in response to a terrorism incident, etc, is that the people whose terminal values are served by that outcome (such as, I am assuming, you) share the universe with people whose terminal values assign a negative value to that outcome (such as me). By using this method of increasing your utility you impose severe negative utility on me.

Note that if you engage in ordinary quantum suicide then my circumstances remain materially the same in both Everett branches, and the objection I just described does not apply.

Richard, I am going to assume ... that you assign an Everett branch in which you painless wink out of existence a value of zero (neither desirable or undesirable)

I'd rather say that people who find quantum suicide desirable have a utility function that does not decompose into a linear combination of individual utility functions for their individual Everett branches-- even if they had to deal with a terrorist attack on all of these branches, say. Surely everybody here would find an outcome undesirable where all of their future Everett branches wink out of existence. So if somebody prefers one Everett branch winking out and one continuing to exist to both continuing to exist, you can only describe their utility function by looking at all the branches, not by looking at the different branches individually. (Did that make sense?)

Gawk! "even if they had to deal with a terrorist attack on all of these branches, say" was supposed to come after "Surely everybody here would find an outcome undesirable where all of their future Everett branches wink out of existence." (The bane of computers. On a typewriter, this would not have happened.)

Did that make sense?

Yes, and I can see why you would rather say it that way.

My theory is that most of those who believe quantum suicide is effective assign negative utility to suffering and also assign a negative utility to death, but knowing that they will continue to live in one Everett branch removes the sting of knowing (and consequently the negative utility of the fact) that they will die in a different Everett branch. I am hoping Cameron Taylor or another commentator who thinks quantum suicide might be effective will let me know whether I have described his utility function.

I assign quantum suicide a utility of "(utility(death) + utility(alternative))/ 2 - time wasted - risk of accidently killing yourself while making the death machine". That is to say, I think it is bloody stupid.

What I do assert is that anyone answering 'yes' to Elizier's proposal to destroy the universe with an LHC to avert terrorism would also be expected to use the same mechanism to achieve any other goal for which the utility is lower than the cost of creating an LHC. For E, that would mean his FAI. The question seems to logically imply one of:
- Eliezer can see something different between LHC death and cyanide death.
- There are some really messed up utility functions out there.
- The question is simply utterly trivial, barely worth asking.

I'd rather say that people who find quantum suicide desirable have a utility function that does not decompose into a linear combination of individual utility functions for their individual Everett branches-- even if they had to deal with a terrorist attack on all of these branches, say. Surely everybody here would find an outcome undesirable where all of their future Everett branches wink out of existence. So if somebody prefers one Everett branch winking out and one continuing to exist to both continuing to exist, you can only describe their utility function by looking at all the branches, not by looking at the different branches individually. (Did that make sense?)
------

I like your explanation Benja. There is no particular reason why a utility function need to consider 'branch winking out of existence' with the same simplicity with which they evaluate more mundane catastrophes. For example, consider the practice of thousands of gamers out there: "That start sucks! Restart!" I can give no mathematical reason why this preference ought to be dismissed.

If it fails 100 times in a row, i`ll sue the researchers for killing me a hundred times in all those other realities.

Oh the humanity-ity-ity-ty-ty-y-y-y-y!

Of course the future repeated failures of the LHC have got to seem non-miraculous though since the likelhood of each experiment failing becomes lower the more experiments you plan on running.

Perhaps some sort of funding problem after a collapse of the world financial system, but that`s not likely, is it?

It`s like the idea applying the idea of quantum immortality and the anthropic principle to my own experience. Wouldn`t it make sense for me to observe my apparent immortality in a world where immortality wasn`t miraculous, such as when technology had advanced to a point where it was `normal`.

A bit of a contradiction there, technology advances to the point where destruction of humanity is easy, but immortality is possible as well.

Benja: Wrong analogy. You left out a bit. All people who actually HAVE CANCER AND that would get a POSITIVE RESULT are killed during the mammograph, never to receive the result. Your task is then to condition on first *receiving a result* and then *that result being positive* and alter your estimate of how likely you are to have cancer.

(Depending on how you meant the analogy it may be the negative result + positive actual cancer who are killed. Point is, your analogy completely misses the point. *Not every person who takes the test gets a result but you do. That is important.*)

Pardon me, ignore that or delete it. I clicked "How Many LHC Failures Is Too Many?" rather than "James" on the recent posts link. Death.

The Anti-LHC Conspiracy strikes again! LHC might not go online until 2010.

Right, that's it, I'm gonna start cooking up some nitroglycerin and book my Eurostar ticket tonight. Who's with me?

I dread to think of the proportion of my selves that have already suffered horrible gravitational death.

The comments to this entry are closed.

## May 2009

Sun Mon Tue Wed Thu Fri Sat
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31