« Break Cryonics Down | Main | Toddlers Avoid Dissenters »

March 17, 2009


"Most AIs gone wrong are just going to disassemble you, not hurt you. I think I've emphasized this a number of times, which is why it's surprising that I've seen both you and Robin Hanson, respectable rationalists both, go on attributing the opposite opinion to me."

My apologies. I went back over some of your writings, sure I would find contradicting evidence, but it seems I am suffering from recall bias: the most horrifying scenarios were just the ones that came to mind most easily.

Yvain wrote: "I know that many of the very elderly people I know claim they're tired of life and just want to die already, and I predict that I have no special immunity to this phenomenon that will let me hold out forever. But I don't know how much of that is caused by literally being bored with what life has to offer already, and how much of it is caused by decrepitude and inability to do interesting things."

I think it definitely has to do with senescence and decrepitude. I bet that in the past when life expectancy was much shorter, people of 50 years of age (in bad shape) felt the same as some 80 years old do now. Nobody likes to suffer, nobody likes impotence. Remove those and that changes a lot of things.

If you had the body and mind of a 30 years old, I doubt you'd feel like that. I expect the universe to be big and varied enough to entertain someone for quite a while. Maybe if everything stayed totally static it could get boring, but I expect arts, science, technology, politics, etc, to keep changing.


"And that's just the complicated failure - the simple one is that the government of Communist China develops the Singularity AI and programs it to do whatever they say."

It's hard to develop an AI that does as you say. It looks like it's easier to develop an AI that does as you want. People in the government of Comminust China are not mutants. They are just like other people. So, if the Communist China develops AGI, it's again more likely to be either a FAI or a Paperclipper AI than an Evil Communist AI.

Vladimir: I don't think the Communists would create an evil AI, but I don't think they'd create an Eliezer-style friendly AI either. I think they'd create an AI that does what they tell it. I don't think such a world would be Hell, but I don't think it would be any better than Communist China today, and it would bear the addition problem that you couldn't circumvent the censors and you'd have no hope of escaping or overthrowing it.

The Chinese wouldn't immediately become evil mutants when creating an AI, but they wouldn't immediately become peace-and-freedom hippies either. Keep in mind that one of the most surprising aspects of the SIAI's plan is that they don't intend to just program it to enact their own values all over the world. It's possible that absolute power mellows people out because they don't have to be so paranoid (see Mencius' post about Fnargl) but I wouldn't count on it.

TGGP: The hedonic set point is good point, but not easy to grok. Taken literally, it would mean that North Korean refugees who flee to South Korea are wasting their time, and that you should be equally willing to move to Burma as to eg the UK. It also implies that fighting to end dictatorship/help the economy/promote good policies is a stupid goal since it doesn't help anyone. I'm still struggling to understand the implications of this for normal everyday morality, but until I do I'd rather not use it for cryonics.

Michael: Good point. I'm currently reconsidering my opposition in light of Eliezer's explanation that he thinks dystopian AI is unlikely.

Everyone: This next argument isn't My Real Objection, and discussing it will have no bearing on whether I sign up for cryonics or not, but I was thinking about it earlier today. Given MWI, I can assume that in some Everett branch I'll probably remain alive no matter what (I can even ensure this by generating a random number and signing up for cryonics if it falls within a specific small range). Although I do identify with my cryonically revived body, I don't identify with it any more than I identify with an identical Yvain from another Everett branch. Doesn't that mean that as long as I don't have a goal of maximizing the number of Yvains in the multiverse, I can satisfy my goal of continuing the existence of a being with whom I identify without signing up for cryonics, or by signing up for cryonics only if a coin comes up heads a hundred times in a row? (one reason this isn't my true objection: it implies that I should be indifferent to committing suicide in the present. I don't know *why* it's wrong, though. And I can't be the first person to think of this.)

No, really. (It sounds like you missed my argument, since you just restated your position in greater detail.) I think it's very hard to create an AI that does as you say. It'd be very hard to create an AI that follows government's orders, without screwing them up to a point of dismantling the world. It looks like a much simpler concept to create an AI that follows the deeper intentions of specified agents, and since those agents are not mutants, the intention should be fine for other people too. So, I expect the China AI to either dismantle the world by mistake, or to be an "Eliezer-style" FAI, and I don't expect an orders-following AGI.

P.S. The argument from MWI suicide is wrong because you care about measure of things, the same way you care about probability in decision theory. You don't want to just win, you also want to win with sufficient probability/measure.

Johnicholas, can you comment on this?

At AGI 2009, Selmer Bringsjord presented a paper, General Intelligence and Hypercomputation, which says:

The mathematics of hypercomputation is now quite developed; the machines, definitions, and theorems in question are elegant and informative (e.g., see (SS94; Sie99; EN02; Cop98; HL00; BKS+06; BZ03) ... It's a brute fact that human cognizers, in the logico-mathematical realm, conceive, manipulate, reason over the space H (F - T ) above what Turing machines and their equivalents can muster. Were this not happening, we would not have the mathematics of hypercomputation summarized above, the first part of which was discovered in 1965, when one of the first hypercomputing machines (trial-and-error machines) were specifi ed (Gol65; Put65). In this activity, the humans in question use formal schemes that cannot even be directly represented in any of the languages Turing machines and their equivalents are restricted to using.

When he says that they "use formal schemes that cannot even be represented", he is obviously wrong, since I assume these schemes were published in books that would still be readable if reduced to bitmaps.

Is there any sense to the rest of his argument? I would be shocked if the answer were yes, but I know nothing about "hypercomputation".

Johnicholas, the set of real numbers does not have a 1:1 mapping onto a set of string names.

For example, what string corresponds to pi?

Great post, Eliezer!

I'm not sure why people suggest that Islam counterbalances against Christianity more than against atheism. It's true that belief in the divinity of Jesus contradicts the tawhid of Allah, and for that reason many Muslims do belief Christians go to hell. But there are also some early suras in the Qur'an suggesting that Christians, as "people of the Book," will be saved (e.g., 2:62, 3:113-15, 3:199, 5:82-85). In contrast, belief in God is a definite requirement for salvation, so Allah would most likely send atheists to hell.

I think the amount of resources (say time and money) you have is crucial. Even with super-large amounts of resources I wouldn't spend time on believing in Christianity for, among others, reasons pointed out in the article.

Having $1000 dollar to spend every month, would I spend $50 of these on cryonics insurance? No! I think it is more rational to invest those money in high-risk stock. Maybe the way to Very Long Lifespan turn out to be uploading, or just continually fixing broken parts of the body, or a any of a range of other possibilities.

Someone might point out that cryonics is available today and thus stands out. Sure, but what if I survive for fifty more years, a rejuvenation technology appears and I cannot afford it?

Private investment groups GEM (Global Emerging Markets) Global Yield Fund Limited and GEM Investment Advisors Inc. had committed investing up to P300 million in local technology firm IPVG Corp. through the purchase of new IPVG shares and shares from existing shareholder Elite Holdings Inc.

The investment, which is for primary as well as secondary shares, involves both the company (IPVG) and one of the principal shareholders (ELITE).

The agreement also provides that IPVG shall issue to GEM or to GEM’s order, one or more warrant(s) to subscribe for up to 30 million shares.

The funding will be used for IPVG’s future business activities and engagements, and for the expansion of its operating subsidiaries.

IPVG CEO Enrique Gonzalez said the investment provides IPVG financing for expansion and for the organic capital requirements of our business subsidiaries. We welcome GEM’s entry into our company as they bring with them a strong track record in private equity and capital markets from their investment activities around the world.

“Despite a challenging global macro environment, this deal is evidence that well run companies can attract smart capital,” Gonzalez closes.

The GEM Group, comprising GEM Investment Advisors Inc. and GEM Global Yield Fund Limited and their affiliates, which was founded in 1991, is a $2.7 billion alternative investment firm engaged in the management of a diverse set of investment tools centered on emerging markets all over the world.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30