The Pascal's Wager Fallacy Fallacy

Today at lunch I was discussing interesting facets of second-order logic, such as the (known) fact that first-order logic cannot, in general, distinguish finite models from infinite models.  The conversation branched out, as such things do, to why you would want a cognitive agent to think about finite numbers that were unboundedly large, as opposed to boundedly large.

So I observed that:

1. Although the laws of physics as we know them don't allow any agent to survive for infinite subjective time (do an unboundedly long sequence of computations), it's possible that our model of physics is mistaken.  (I go into some detail on this possibility below the cutoff.)
2. If it is possible for an agent - or, say, the human species - to have an infinite future, and you cut yourself off from that infinite future and end up stuck in a future that is merely very large, this one mistake outweighs all the finite mistakes you made over the course of your existence.

And the one said, "Isn't that a form of Pascal's Wager?"

I'm going to call this the Pascal's Wager Fallacy Fallacy.

You see it all the time in discussion of cryonics.  The one says, "If cryonics works, then the payoff could be, say, at least a thousand additional years of life."  And the other one says, "Isn't that a form of Pascal's Wager?"

The original problem with Pascal's Wager is not that the purported payoff is large.  This is not where the flaw in the reasoning comes from.  That is not the problematic step.  The problem with Pascal's original Wager is that the probability is exponentially tiny (in the complexity of the Christian God) and that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God).

However, what we have here is the term "Pascal's Wager" being applied solely because the payoff being considered is large - the reasoning being perceptually recognized as an instance of "the Pascal's Wager fallacy" as soon as someone mentions a big payoff - without any attention being given to whether the probabilities are in fact small or whether counterbalancing anti-payoffs exist.

And then, once the reasoning is perceptually recognized as an instance of "the Pascal's Wager fallacy", the other characteristics of the fallacy are automatically inferred: they assume that the probability is tiny and that the scenario has no specific support apart from the payoff.

But infinite physics and cryonics are both possibilities that, leaving their payoffs entirely aside, get significant chunks of probability mass purely on merit.

Yet instead we have reasoning that runs like this:

1. Cryonics has a large payoff;
2. Therefore, the argument carries even if the probability is tiny;
3. Therefore, the probability is tiny;
4. Therefore, why bother thinking about it?
   

(Posted here instead of Less Wrong, at least for now, because of the Hanson/Cowen debate on cryonics.)

Further details:

Pascal's Wager is actually a serious problem for those of us who want to use Kolmogorov complexity as an Occam prior, because the size of even the finite computations blows up much faster than their probability diminishes (see here).

See Bostrom on infinite ethics for how much worse things get if you allow non-halting Turing machines.

In our current model of physics, time is infinite, and so the collection of real things is infinite.  Each time state has a successor state, and there's no particular assertion that time returns to the starting point.  Considering time's continuity just makes it worse - now we have an uncountable set of real things!

But current physics also says that any finite amount of matter can only do a finite amount of computation, and the universe is expanding too fast for us to collect an infinite amount of matter.  We cannot, on the face of things, expect to think an unboundedly long sequence of thoughts.

The laws of physics cannot be easily modified to permit immortality: lightspeed limits and an expanding universe and holographic limits on quantum entanglement and so on all make it inconvenient to say the least.

On the other hand, many computationally simple laws of physics, like the laws of Conway's Life, permit indefinitely running Turing machines to be encoded.  So we can't say that it requires a complex miracle for us to confront the prospect of unboundedly long-lived, unboundedly large civilizations.  Just there being a lot more to discover about physics - say, one more discovery of the size of quantum mechanics or Special Relativity - might be enough to knock (our model of) physics out of the region that corresponds to "You can only run boundedly large Turing machines".

So while we have no particular reason to expect physics to allow unbounded computation, it's not a small, special, unjustifiably singled-out possibility like the Christian God; it's a large region of what various possible physical laws will allow.

And cryonics, of course, is the default extrapolation from known neuroscience: if memories are stored the way we now think, and cryonics organizations are not disturbed by any particular catastrophe, and technology goes on advancing toward the physical limits, then it is possible to revive a cryonics patient (and yes you are the same person).  There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities.

Comments

[Johnicholas]

You reference a popular idea, something like "The integers are countable, but the real number line is uncountable." I apologize for nitpicking, but I want to argue against philosophers (that's you, Eliezer) blindly repeating this claim, as if it was obvious or uncontroversial.

Yes, it is strictly correct according to current definitions. However, there was a time when people were striving to find the "correct" definition of the real number line. What people ended up with was not the only possibility, and Dedekind cuts (or various other things) are a pretty ugly, arbitrary construction.

The set containing EVERY number that you might, even in principle, name or pick out with a definition is countable (because the set of names, or definitions, is a subset of the set of strings, which is countable).

The Lowenheim-Skolem theorem says (loosely interpreted) that even if you CLAIM to be talking about uncountably infinite things, there's a perfectly self-consistent interpretation of your talk that refers only to finite things (e.g. your definitions and proofs themselves).

You don't get magical powers of infinity just from claiming to have them. Standard mathematical talk is REALLY WEIRD from a computer science perspective.

[steven]

There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities.

Expected utility is the product of two things, probability and utility. Saying the probability is smaller is not a complete argument.

[Yvain]

"There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities."

That doesn't seem at all obvious to me. First, our current society doesn't allow people to die, although today law enforcement is spotty enough that they can't really prevent it. I assume far future societies will have excellent law enforcement, including mind reading and total surveillance (unless libertarians seriously get their act together in the next hundred years). I don't see any reason why the taboo on suicide *must* disappear. And any society advanced enough to revive me has by definition conquered death, so I can't just wait it out and die of old age. I place about 50% odds on not being able to die again after I get out.

I'm also less confident the future wouldn't be a dystopia. Even in the best case scenario the future's going to be scary through sheer cultural drift (see: legalized rape in Three Worlds Collide). I don't have to tell *you* that it's easier to get a Singularity that goes horribly wrong than one that goes just right, and even if we restrict the possibilities to those where I get revived instead of turned into paperclips, they could still be pretty grim (what about some well-intentioned person hard-coding in "Promote and protect human life" to an otherwise poorly designed AI, and ending up with something that resurrects the cryopreserved...and then locks them in little boxes for all eternity so they don't consume unnecessary resources.) And then there's just the standard fears of some dictator or fundamentalist theocracy, only this time armed with mind control and total surveillance so there's no chance of overthrowing them.

The deal-breaker is that I really, really don't want to live forever. I might enjoy living a thousand years, but not forever. You could change my mind if you had a utopian post-singularity society that completely mastered Fun Theory. But when I compare the horrible possibility of being forced to live forever either in a dystopia or in a world no better or worse than our own, to the good possibility of getting to live between thousand years and forever in a Fun Theory utopia that can keep me occupied...well, the former seems both more probable and more extreme.

[Carl Shulman]

"that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God)."
Utilitarian would rightly attack this, since the probabilities almost certainly won't wind up exactly balancing. A better argument is that wasting time thinking about Christianity will distract you from more probable weird-physics and Simulation Hypothesis Wagers.

A more important criticism is that humans just physiologically don't have any emotions that scale linearly. To the extent that we approximate utility functions, we approximate ones with bounded utility, although utilitarians have a bounded concern with acting or aspiring to act or believing that they aspire to act as though they have concern with good consequences that is close to linear with the consequences, i.e. they have a bounded interest in 'shutting up and multiplying.'

[Anonymous]

Johnicholas:

I agree with your sentiment, however:

There is a perfectly good description of the real numbers that is not ugly. Namely, the real numbers are a complete Archimedean ordered field.

To actually construct them, I think using (Cauchy) convergent sequences of rational numbers would be much less ugly than using Dedekind cuts.

Also, the Löwenheim–Skolem theorem only applies to first-order logic, not second-order logic. Why are you constraining me to use only first-order logic? You have to explain that first.

[Larry D'Anna]

"first-order logic cannot, in general, distinguish finite models from infinite models."

Specifically, if a fist order theory had arbitrarily large finite models, then it has an infinite one.

[Eliezer Yudkowsky]

There is no first-order sentence which is true in all and only finite models and not in any infinite models.

Sketch of conventional proof: The compactness theorem says that if a collection of first-order sentences is inconsistent, then a finite subset of those first-order sentences is inconsistent.

To a sentence or theory true of all finite sets, adjoin the infinite series of statements "This model has at least one element", "This model has at least two elements" (that is, there exist a and b with a != b), "This model has at least three elements" (the finite sentence: exists a, b, c, and a != b, b != c, a != c), and so on.

No finite subset of these statements is inconsistent with the original theory, therefore by compactness the set as a whole is consistent with the original theory. Therefore the original theory possesses an infinite model. QED.

[Michael G.R.]

Yvain wrote: "The deal-breaker is that I really, really don't want to live forever. I might enjoy living a thousand years, but not forever. "

I'm curious to know how you know that in advance? Isn't it like a kid making a binding decision on its future self?

As Aubrey says, (I'm paraphrasing): "If I'm healthy today and enjoying my life, I'll want to wake up tomorrow. And so on." You live a very long time one day at a time.

[Anna Salamon]

The “isn’t that like Pascal’s wager?” response is plausibly an instance of dark side epistemology, and one that affects many aspiring rationalists.

Many of us came up against the Pascal’s wager argument at some point before we gained much rationality skill, disliked the conclusion, and hunted around for some means of disagreeing with its reasoning. The overcomingbias thread discussing Pascal’s wager strikes me as including a fair number of fallacious comments aimed at finding some rationale, any rationale, for dismissing Pascal’s wager.

If these arguments tended merely to be about factual matters (“Pascal’s wager can’t be true, because, um, the moon moves in such-and-such a manner”), attempts to dismiss Pascal’s wager without solid argument would perhaps not be all that problematic . But in the specific case of Pascal’s wager, ad hoc rationalizations for its dismissal tend to center around methodological claims: people dislike the conclusion, and so make claims against expected value calculations, expected value calculations’ applicability to high payoffs, or other inference or decision theoretic methodologies. This is exactly dark side epistemology; someone dislikes a conclusion, does not have a principled derivation of why the conclusion should be false, and so seizes on some methodology or other to bolster their dismissal -- and then imports that methodology into the rest of their life, with harmful consequences (e.g., avoiding cryonics).

I’m not endorsing Pascal’s wager. Carl’s critique (above, and also in the original thread) strikes me as valid. It’s just that we need to be really really careful about making up rationalizations and importing those rationalized methodologies elsewhere; the rationalizations can hurt us even in cases where the rationalized conclusion turns out to be true. And Pascal’s wager is such an easy situation in which to make methodological rationalizations -- it sounds absurd, it involves religion, which is definitely uncool, and its claimed conclusion threatens things many of us care about, such as how we live our lives and form beliefs. We might need “be specially on guard!” routines for such situations.

[Carl Shulman]

The fallacious arguments against Pascal's Wager are usually followed by
motivated stopping.

[Pablo Stafforini]

Utilitarian would rightly attack this, since the probabilities almost certainly won't wind up exactly balancing.

Utilitarian's reply seems to assume that probability assignments are always precise. We may plausibly suppose, however, that belief states are sometimes vague. Granted this supposition, we cannot infer that one probability is higher than the other from the fact that probabilities do now wind up exactly balancing.

[Carl Shulman]

Pablo,

Vagueness might leave you unable to subjectively distinguish probabilities, but you would still expect that an idealized reasoner using Solomonoff induction with unbounded computing power and your sensory info would not view the probabilities as exactly balancing, which would give infinite information value to further study of the question.

The idea that further study wouldn't unbalance estimates in humans is both empirically false in the cases of a number of smart people who have undertaken it, and looks like another rationalization.

[g]

Eliezer, it seems to me that you may be being unfair to those who respond "Isn't that a form of Pascal's wager?". In an exchange of the form

Cryonics Advocate: "The payoff could be a thousand extra years of life or more!"

Cryonics Skeptic: "Isn't that a form of Pascal's wager?"

I observe that CA has made handwavy claims about the size of the payoff, hasn't said anything about how the utility of a long life depends on its length (there could well be diminishing returns), and hasn't offered anything at all like a probability calculation, and has entirely neglected the downsides (I think Yvain makes a decent case that they aren't obviously dominated by the upside). So, here as in the original Pascal's wager, we have someone arguing "put a substantial chunk of your resources into X, which has uncertain future payoff Y" on the basis that Y is obviously very large, and apparently ignoring the three key subtleties, namely how to get from Y to the utility-if-it-works, what other low-probability but high-utility-delta possibilities there are, and just what the probability-that-it-works is. And, here as with the original wager, if the argument does work then its consequences are counterintuitive to many people (presumably including CS).

That wouldn't justify saying "That is just Pascal's wager, and I'm not going to listen to you any more." But what CS actually says is "Isn't that a form of Pascal's wager?". It doesn't seem to me an unreasonable question, and it gives CA an opportunity to explain why s/he thinks the utility really is very large, the probability not very small, etc.

I think the same goes for your infinite-physics argument.

I don't see any grounds for assuming (or even thinking it likely) that someone who says "Isn't that just a form of Pascal's wager?" has made the bizarrely broken argument you suggest that they have. If they've made a mistake, it's in misunderstanding (or failing to listen to, or not guessing correctly) just what the person they're talking to is arguing.

Therefore: I think you've committed a Pascal's Wager Fallacy Fallacy Fallacy.

[Carl Shulman]

g,

This is based on the diavlog with Tyler Cowen, who did explicitly say that decision theory and other standard methodologies doesn't apply well to Pascalian cases.

[Vladimir Nesov]

@Yvain:
Don't look at the future as containing you, ask what can the future do worse or better, if it's in possession of the information about you. It can reconstruct you-alive using that information, and let the future you enjoy the life in the future, or it could reconstruct you-alive and torture it for eternity. But in which of these cases the future will actually get better or worse, depending on whether you give the future the information about your structure? Is the torture-people future going to get better because you don't give them specifically the information about your brain? That torture-people future must be specifically evil if it cares so much about creating torture experience especially for the real people who lived in the past, as opposed to, say, paperclipping the universe with the torture chambers full of randomly generated people. Evil is far harder than good or mu, you have to get the future almost right for it to care about people at all, but somehow introduce a sustainable evil twist to it.

[mathusalem]

these posts are useful to calibrate the commitment and self incentive biases. based on the probabilities espoused (80%, bad outcomes are 'exotic') i say the impact is 1000x. the world looks pretty utopian from the a/c cooled academics labs in US in anno domini 2009.

[Alex M]

My question is very specific, can you elaborate on what you mean by "holographic limits on quantum entanglement"? I did a search but all I got was woo-woo websites.

Thank you.

[steven]

Vladimir, hell is only one bit away from heaven (minus sign in the utility function). I would hope though that any prospective heaven-instigators can find ways to somehow be intrinsically safe wrt this problem.

[Vladimir Nesov]

Steven, even the minus-utility hell won't get worse because it has information useful for the positive-utility eutopia. Only and specifically the positive-utility eutopia could have a use for such information. You win from providing this information in case of a good outcome, and you don't lose in case of a bad outcome.

[Eliezer Yudkowsky]

Alex, google: "Holographic bound".

[g]

Carl, it clearly isn't based *only* on that since Eliezer says "You see it all the time in discussion of cryonics".

[Alex M]

Eliezer, thanks I've found material on the holographic principle and did some reading myself. it's an intriguing idea, but an idea so far that has no experimental basis yet. Aside from unconfirmed source of noise in a gravitational wave experiment, it's not known if holographic principle/cosmological information bound actually plays a role. Why did you include that in your post, were you just including another possible example of how universe seems to conspire against our ambitions.

[Dmitriy Kropivnitskiy]

Pascal Wager != Pascal Wager Fallacy. If original Pascal wager didn't depend on a highly improbable proposition (existence of a particular version of god), it would be logically sound (or at least more sound then it is). So, I don't see a problem comparing cryonics advocacy logic with Pascal's wager.

On the other hand, I find some of the probability estimates cryonics advocates make to be unsound, so for me, this way of cryonics advocacy does look like a Pascal Wager Fallacy. In particular, I don't see why cryonics advocates put high probability values on being revived in the future (number 3 in Robert Hanson's post) and liking the future enough to want to live there (look at Yvain's comment to this post). Also, putting unconditional high utility value on long life span seems to be a doubtful proposition. I am not sure that life of torture is better than non-existence.

[Doug S.]

What if we phrase a Pascal's Wager-like problem like this:

If every winner of a certain lottery receives $300 million, a ticket costs $1, the chances of winning are 1 in 250 million, and you can only buy one ticket, would you buy that ticket?

There's a positive expected value in dollars, but 1 in 250 million is basically not gonna happen (to you, at least).

[vroman]

@ doug S

I defeat your version of the PW by asserting there is no rational lottery operator who goes forth with the business plan to straight up lose $50million. thus the probability of your scenario, as w the christian god, is zero.

[g]

vroman, see the post on Less Wrong about least-convenient possible worlds. And the analogue in Doug's scenario of the existence of (Pascal's) God isn't the reality of the lottery he proposes -- he's just asking you to accept that for the sake of argument -- but your winning the lottery.

[Nick Tarleton]

I think a heuristic something like this is often involved: "If someone claims a high benefit (at any probability) for some costly implausible course of action, there's a good chance they're (a) consciously trying to exploit me, (b) infected by a parasitic meme, or (c) getting off on the delusion that they have a valuable Cause. In any of those cases, they'll probably have plenty of persuasive invalid arguments; if I try to analyze these, I may be convinced in spite of myself, so I'd better find whatever justification I can to stop thinking."

vroman: See The Least Convenient Possible World.

Carl: Islam and Christianity may not balance, but what about Christianity and anti-Christianity?

[Doug S.]

vroman: Two words - rollover jackpots.

[vroman]

I read and understood the Least convenient possible world post.
given that, then let me rephrase your scenario slightly

If every winner of a certain lottery receives $X * 300 million, a ticket costs $X, the chances of winning are 1 in 250 million, you can only buy one ticket, and $X represents an amount of money you would be uncomfortable to lose, would you buy that ticket?

answer no. If the ticket price crosses a certain threshold, then I become risk averse. if it were $1 or some other relatively inconsequential amount of money, then I would be rationally compelled to buy the nearly-sure loss ticket.

[Carl Shulman]

Nick,

"Islam and Christianity may not balance, but what about Christianity and anti-Christianity?"
Why would you think that Christianity and anti-Christianity plausibly balance exactly? Spend some time thinking about the distribution of evolved minds and what they might simulate, and you'll get divergence.

[Nick Tarleton]

Why would you think that Christianity and anti-Christianity plausibly balance exactly?

Because I've been thinking about algorithmic complexity, not the actions of agents. Good point.

[Nick Tarleton]

Specifically, thinking of the algorithmic complexity of the religion - if I were to use priors here, I should be thinking about utility(belief)*prior probability of algorithms computing functions from beliefs to reward or punishment.

[Jay]

Ask yourself if you would want to revive someone frozen 100 years ago. Most Americans of the time were unabashedly racist, had little concept of electricity and none of computing, had vaguely heard of automobiles, etc. They'd be awakened into a world that they don't understand, a world that judges them by mysterious criteria. It would be worse than being foreign, because the new culture's values were formed at least partially in reaction to the perceived problems of the past.

[Eliezer Yudkowsky]

Ask yourself if you would want to revive someone frozen 100 years ago.

Yes. They don't deserve to die. Kthx next.

[Benja Fallenstein]

Ask yourself if you would want to revive someone frozen 100 years ago.

Yes. They don't deserve to die. Kthx next.

I wish that this were on Less Wrong, so that I could vote this up.

[steven]

Does nobody want to address the "how do we know U(utopia) - U(oblivion) is of the same order of magnitude as U(oblivion) - U(dystopia)" argument? (I hesitate to bring this up in the context of cryonics, because it applies to a lot of other things and because people might be more than averagely emotionally motivated to argue for the conclusion that supports their cryonics opinion, but you guys are better than that, right? right?)

Carl, I believe the point is that until I know of a specific argument why one is more likely than the other, I have no choice but to set the probability of christianity equal to the probability of anti-christianity, even though I don't doubt such arguments exist. (Both irrationality-punishers and immorality-punishers seem far less unlikely than nonchristianity-punishers, so it's moot as far as I can tell.)

Vladimir, your argument doesn't apply to moralities with an egoist component of some sort, which is surely what we were discussing even though I'd agree they can't be justified philosophically.

I stand by all the arguments I gave against Pascal's wager in the comments to Utilitarian's post, I think.

[Michael G.R.]

"Most Americans of the time were unabashedly racist, had little concept of electricity and none of computing, had vaguely heard of automobiles, etc."

So if you woke up in a strange world with technologies you don't understand (at first) and mainstream values you disagree with (at first), you would rather commit suicide than try to learn about this new world and see if you can have a pleasant life in it?

[Carl Shulman]

Steven,

Information value.

[Nick Tarleton]

irrationality-punishers and immorality-punishers seem far less unlikely than nonchristianity-punishers

If you mean "in rough proportion to the algorithmic complexity of Christianity", nonmajoritarianism-punishers, and presumably plenty of other simple entities, would effectively be nonchristianity-punishers. Probably still true, though.

[Vladimir Nesov]

Steven, to account for the especially egoist morality, all you need to do is especially value future-you. I don't see how it changes my points.

[Eliezer Yudkowsky]

Nick, Christians are not a majority (and if they were, an alternative course would be to try to shift majority opinions to something easier to believe, preferably before you died but it has to get done...)

I'm not claiming that U(utopia) - U(oblivion) ~ U(oblivion) - U(dystopia + revival + no suicide), but the question is whether the factor describing the relative interval, is greater than the factor of diminished probability for U(dystopia + revival + no suicide), which seems large. Also, steven points out for the benefit of altruists that if it's not you who's tortured in the future dystopia, the same resources will probably be used to create and torture someone else.

Though I hesitate to point this out, the same logic against cryonic suspension also implies that egoists, but not altruists, should immediately commit suicide in case someone is finishing their AI project in a basement, right now. A good number of arguments against cryonics also imply suicide in the present.

[Yvain]

"I'm curious to know how you know that in advance? Isn't it like a kid making a binding decision on its future self? As Aubrey says, (I'm paraphrasing): "If I'm healthy today and enjoying my life, I'll want to wake up tomorrow. And so on." You live a very long time one day at a time."

Good point. I usually trust myself to make predictions of this sort. For example, I predict that I would not want to eat pizza every day in a row for a year, even though I currently like pizza, and this sort of prediction has worked in the past. But I should probably think harder before I become certain that I can make this prediction with something more complicated like my life. I know that many of the very elderly people I know claim they're tired of life and just want to die already, and I predict that I have no special immunity to this phenomenon that will let me hold out forever. But I don't know how much of that is caused by literally being bored with what life has to offer already, and how much of it is caused by decrepitude and inability to do interesting things.

"Evil is far harder than good or mu, you have to get the future almost right for it to care about people at all, but somehow introduce a sustainable evil twist to it."

In all of human society-space, not just the ones that have existed but every possible combination of social structures that could exist, I interpret only a vanishingly small number (the ones that contain large amounts of freedom, for example) as non-evil. Looking over all of human history, the number of societies I would have enjoyed living in are pretty minimal. I'm not just talking about Dante's Hell here. Even modern day Burma/Saudi Arabia, or Orwell's Oceania would be awful enough to make me regret not dying when I had the chance.

I don't think it's so hard to get a Singularity that leaves people alive but is still awful. If the problem is a programmer who tried to give it a sense of morality but ended up using a fake utility function or just plain screwing up, he might well end with a With Folded Hands scenario or Parfit's Mere Addition Paradox (I remember Eliezer saying once - imagine if we get an AI that understands everything perfectly except freedom) . And that's just the complicated failure - the simple one is that the government of Communist China develops the Singularity AI and programs it to do whatever they say.

"Also, steven points out for the benefit of altruists that if it's not you who's tortured in the future dystopia, the same resources will probably be used to create and torture someone else."

I think that's false. In most cases I imagine, torturing people is not the terminal value of the dystopia, just something they do to people who happen to be around. In a pre-singularity dystopia, it will be a means of control and they won't have the resources to 'create' people anyway, (except the old-fashioned way). In a post-singularity dystopia, resources won't much matter and the AI's more likely to be stuck under injunctions to protect existing people than trying to create new ones (unless the problem is the Mere Addition Paradox). Though I admit it would be a very specific subset of rogue AIs that view frozen heads as "existing people".

"Though I hesitate to point this out, the same logic against cryonic suspension also implies that egoists, but not altruists, should immediately commit suicide in case someone is finishing their AI project in a basement, right now. A good number of arguments against cryonics also imply suicide in the present."

I'm glad you hesitated to point it out. Luckily, I'm not as rationalist as I like to pretend :) More seriously, I currently have a lot of things preventing me from suicide. I have a family, a debt to society to pay off, and the ability to funnel enough money to various good causes to shape the future myself instead of passively experience it. And less rationally but still powerfully, I have the self-preservation urge pretty strongly that would probably kick in if I tried anything. Someday when the Singularity seems very near, I really am going to have to think about this more closely. If I think a dictator's about to succeed on an AI project, or if I've heard about the specifics of the a project's code and the moral system seems likely to collapse, I do think I'd be sitting there with a gun to my head and my finger on the trigger.

[Yvain]

One more thing: Eliezer, I'm surprised to be on the opposite side as you here, because it's your writings that convinced me a catastrophic singularity, even one from the small subset of catastrophic singularities that keep people alive, is so much more likely than a good singularity. If you tell me I'm misinterpreting you, and you assign high probability to the singularity going well, I'll update my opinion (also, would the high probability be solely due to the SIAI, or do you think there's a decent chance of things going well even if your own project fails?)

[steven]

Nick, I'm now sitting here being inappropriately amused at the idea of Hal Finney as Dark Lord of the Matrix.

Eliezer, thanks for responding to that. I'm never sure how much to bring up this sort of morbid stuff. I agree as to what the question is.

Also, steven points out for the benefit of altruists that if it's not you who's tortured in the future dystopia, the same resources will probably be used to create and torture someone else.

It was Vladimir who pointed that out, I just said it doesn't apply to egoists. I actually don't agree that it applies to altruists either; presumably most anything that cared that much about torturing newly created people would also use cryonauts for raw materials. Also, maybe there are "people who are still alive" considerations.

[ad]

I don't have to tell *you* that it's easier to get a Singularity that goes horribly wrong than one that goes just right

Don't the acceleration-of-history arguments suggest that there will be another singularity, a century or so after the next one? And another one shortly after that, etc?

What are the chances that they will all go exactly right for us?

[Nick Tarleton]

If the problem is a programmer who tried to give it a sense of morality but ended up using a fake utility function or just plain screwing up, he might well end with a With Folded Hands scenario or Parfit's Mere Addition Paradox (I remember Eliezer saying once - imagine if we get an AI that understands everything perfectly except freedom) . And that's just the complicated failure - the simple one is that the government of Communist China develops the Singularity AI and programs it to do whatever they say.

For whatever relief it's worth, someone who thought that was a good idea would have a good chance of building a paperclipper instead. "There is a limit to how competent you can be, and still be that stupid."

[Eliezer Yudkowsky]

Yvain, while it's hard to get a feel on what exactly happens when one of the meddling dabblers tries to give their AI a goal system, I would mostly expect those AIs to end up as paperclip maximizers, or at most, tiling the universe with tiny molecular smiley-faces. Nothing sentient.

Most AIs gone wrong are just going to dissassemble you, not hurt you. I think I've emphasized this a number of times, which is why it's surprising that I've seen both you and Robin Hanson, respectable rationalists both, go on attributing the opposite opinion to me.

[steven]

Eliezer, "more AIs are in the hurting class than in the disassembling class" is a distinct claim from "more AIs are in the hurting class than in the successful class", which is the one I interpreted Yvain as attributing to you.

[retired urologist]

Isn't there already a good deal of experience regarding the attitudes/actions of the most intelligent entity known (in current times, humans) towards cryonically suspended potential sentient beings (frozen embryos)?

[TGGP]

Yvain, people seem to have a hedonic set point. If you currently prefer life to non-life, I highly doubt you would not if you lived in Saudi Arabia or Burma.