« Cowen-Hanson Bloggingheads Topics? | Main | Fake Fish »

August 29, 2008

Comments

the dreaded Maximum Fun Device

I'm going to start watching for opportunities to use this phrase. "Nobody expects the dreaded Maximum Fun Device!"

"In fact, if you're interested in the field, you should probably try counting the ways yourself, before I continue. And score yourself on how deeply you stated a problem, not just the number of specific cases."

I got #1, but I mushed #2 and #3 together into "The AI will rewire our brains into computationally cheap super-happy programs with humanesque neurology", as I was thinking of failure modes and not reasons for why failure modes would be bad.

Will we ever be able to tell whether we've protected everything of value, before the end of an AI's infrahuman stage? We don't do a wonderful job of protecting everything anyway – maybe it's more important to protect what you can.

"Would you have classified the happiness of cocaine as 'happiness', if someone had asked you in another context?"

I'm not sure I understand what you mean here. Do you think it's clear that coke-happiness is happiness, or do you think it's clear that coke-happiness is not happiness?

You're forgetting the philosopher's dotted-line trick of making it clearer by saying it in a foreign language. "Oh, you thought I meant 'happiness' which is ill-defined? I actually meant 'eudaimonia'!"

I can't bring myself to feel sad about not knowing of a disaster that I can't possibly avert.

Nevertheless, I don't get why people would propose any design that is not better than CEV in any obvious way.

But I have a question about CEV. Among the parameters of the extrapolation, there is "growing up closer together". I can't decipher what that means, particularly in a way that makes it a good thing. If it means that I would have more empathy, that is subsumed by "know more". My initial reaction, though, was "my fingers would be closer to your throat".

"This is why, when you encounter the AGI wannabe who hasn't planned out a whole technical approach to FAI, and confront them with the problem for the first time, and they say, "Oh, we'll test it to make sure that doesn't happen, and if any problem like that turns up we'll correct it, now let me get back to the part of the problem that really interests me," know then that this one has not yet leveled up high enough to have interesting opinions."

There is an overwhelming assumption here in a Terminator series hard takeoff. Whereas the plausible reality IMO seems to be more like an ecosystem of groups of intelligences of varying degrees all of which will likely have survival rationale for disallowing a peer to hit nutso escape velocity. And at any rate, someone in 2025 with a infrahuman intelligent AI is likely to be much better off at solving the 100th meta-derivative of these toy problems than someone working with 200 hz neurons alone. Now I gotta go, I think windows needs to reboot or something..

Is intelligence general or not? If it is, then an entity that can do molecular engineering but be completely naive about what humans want it *impossible.*

Aron: """Whereas the plausible reality IMO seems to be more like an ecosystem of groups of intelligences of varying degrees all of which will likely have survival rationale for disallowing a peer to hit nutso escape velocity."""

What can an infrahuman AI do to a superhuman AI?

What can a human do to a superhuman AI that a human + infrahuman AI can't do?

Ian: the issue isn't whether it could determine what humans want, but whether it would care. That's what Eliezer was talking about with the "difference between chess pieces on white squares and chess pieces on black squares" analogy. There are infinitely many computable quantities that don't affect your utility function at all. The important job in FAI is determining how to create an intelligence that will care about the things we care about.

Certainly it's necessary for such an intelligence to be able to compute it, but it's certainly not sufficient.

"If everyone in the universe just ends up with their brains hotwired to experience maximum happiness forever, or perhaps just replaced with orgasmium gloop, is that the greatest possible fulfillment of humanity's potential? Is this what we wish to make of ourselves?"

Why not just bite the bullet and say yes?

Also, your diagrams would hold more weight if you hadn't used Comic Sans.

What on Earth could someone possibly be thinking, when they propose creating a superintelligence whose behaviors are reinforced by human smiles?

Wiring almost anything about today's humans into a proposed long-term utility function seems rather obviously dangerous. Keeping unmodified humans around would cripple our civilisation's spaceworthyness - and would probably eventually lead to our distant descendants being smothered by aliens who didn't have such a limited and backwards notion of what's actually important.

Humans should probably not be allowed to persist for too long past their sell-by date - or they may fill the whole planet with the foul stench of their excrement. Fortunately, it doesn't seem like a likely outcome. Our main asset is our brains - and, fairly soon, that asset will be practically worthless.

Stephen: "the issue isn't whether it could determine what humans want, but whether it would care."

That is certainly an issue, but I think in this post and in _Magical Categories_, EY is leaving that aside for the moment, and simply focussing on whether we can hope to communicate what we want to the AI in the first place.

It seems to me that today's computers are 100% literal and naive, and EY imagines a superintelligent computer still retaining that property, but would it?

Ian C. is implying an AI design where the AI would look at it's programmers and determine what they *really* had wanted to program the AI to do - and would have if they were perfect programmers - and then do that.

But that very function would have to be programmed into the AI. My head, for one, spins in self-refential confusion over this idea.

I for one do not see the benefit in getting superintelligences to follow the preferences of lesser, evolved intelligences. There's no particular reason to believe that humans with access to the power granted by a superintelligence would make better choices than the superintelligence at all - or that their preferences would lead to states that their preferences would approve of, much less actually benefit them.

Just like you wouldn't want an AI to optimize for only some of the humans, you wouldn't want an AI to optimize for only some of the values. And, as I keep emphasizing for exactly this reason, we've got a lot of values.

What if the AI emulates some/many/all human brains in order to get a complete list of our values? It could design its own value system better than any human.

@Caledonian: So if an AI wants to wipe out the human race we should be happy about it? What if it wants to treat as cattle? Which/whose preferences *should* it follow? (Notice the weasel words?)

When I was a teenager I used to think just like you. A superintelligence would have better goals than ordinary humans, _because_ it is superintelligent. Then I grew up and realized that minds are not blank slates, and you can't just create a "value-free" AI and see what kinds of terminal values it chooses for itself.

It might be fruitful to consider related questions: How can humanity ensure that its distant descendants effectively pursue current values? How does a nation ensure that in several generations its government will pursue current values. How can a non-profit organization ensure that its future incarnations pursue the values of its founders? How do parents ensure that their children will pursue parental values? How do individuals ensure that when older they will pursue younger-version values?

There clearly are limited degrees to which all of these things are possible, but it also seems pretty clear the degree is quite limited.

@Caledonian: So if an AI wants to wipe out the human race we should be happy about it? What if it wants to treat as cattle?
'Happiness' is not the point.

And humans already do a great job of treating each other like cattle... possibly because most of us are, one way or another.

AFAICS, the proposal is to build a single entity, that never dies, and faces no competition - and so is not subject to natural selection - and is engineered in order to preserve its aims.

We haven't seen such entities before. Animals act to preserve their aims (gene propagation) - but are pretty primitive, and can get flummoxed in unfamiliar environments. Religions, companies and organisations are typically larger and smarter - but they compete, face natural selection, and are moulded by it. Consequently their aims tend to converge on whatever results in their inheritance persisting.

Caledonian, if you want to build an AI that locks the human race in tiny pens until it gets around to slaughtering us, that's... lovely, and I wish you... the best of luck, but I think all else equal I would rather support the guy who wants to build an AI that saves the world and makes everything maximally wonderful.

Everything being maximally wonderful is a bit like what the birds of paradise have. What will happen when their ecosystem is invaded by organisms which have evolved along less idyllic lines? It is not hard to guess the answer to that one. As always, be careful what you wish for.

Tim Tyler: Wiring almost anything about today's humans into a proposed long-term utility function seems rather obviously dangerous.

Gee, who's articulating this judgment? A fish? A leaf of grass? A rock? Why, no, it's a human!

Caledonian: I for one do not see the benefit in getting superintelligences to follow the preferences of lesser, evolved intelligences. There's no particular reason to believe that humans with access to the power granted by a superintelligence would make better choices than the superintelligence at all - or that their preferences would lead to states that their preferences would approve of, much less actually benefit them.

Who is it that's using these words "benefit", talking about "lesser" intelligences, invoking this mysterious property of "better"-ness? Is it a star, a mountain, an atom? Why no, it's a human!

...I'm seriously starting to wonder if some people just lack the reflective gear required to abstract over their background frameworks. All this talk of moral "danger" and things "better" than us, is the execution of a computation embodied in humans, nowhere else, and if you want an AI that follows the thought and cares about it, it will have to mirror humans in that sense.

"Everything being maximally wonderful is a bit like what the birds of paradise have. What will happen when their ecosystem is invaded by organisms which have evolved along less idyllic lines?"

We win, because anything less would not be maximally wonderful.

Re: We win, because anything less would not be maximally wonderful.

Um, it depends. If we have AI, and they have AI and they have chosen a utility function closer to that which would be favoured by natural selection under such circumstances - then we might well lose.

Is spending the hundred million years gearing up for alien contact - to avoid being obliterated by it - "maximally wonderful"? Probably not for any humans involved. If the preparations are at all serious, ditching the crummy, evolved, backwards human phenotype, and replacing it with a more universe-worthy solution, is likely to be somewhere around step 1.

Tim,

Let's assume that the convergent utility function supported by natural selection is that of a pure survival machine (although it's difficult to parse this, since the entities you're talking about seem indifferent to completely replacing all of their distinguishing features), stripped of any non-survival values of the entity's ancestors. In other words, there's no substantive difference between the survival-oriented alien invaders and human-built survival machines, so why bother to pre-emptively institute the outcome of invasion? Instead we could pursue what we conclude, on reflection is good, trading off between consumption and investment (including investments in expansion and defense) so as to maximize utility. If a modestly increased risk of destruction by aliens is compensated for by much greater achievement of our aims, why should we instead abandon our aims in order to create exactly the outcome we supposedly fear?

Re: All this talk of moral "danger" and things "better" than us, is the execution of a computation embodied in humans, nowhere else, and if you want an AI that follows the thought and cares about it, it will have to mirror humans in that sense.

"Better" - in the sense of competitive exclusion via natural selection. "Dangerous" - in that it might lead to the near-complete obliteration of our biosphere's inheritance. No other moral overtones implied.

AI will likely want to avoid being obliterated - and will want to consume resources, and use them to expand its domain. It will share these properties with all living systems - not just us. It doesn't need to do much "mirroring" to acquire these attributes - they are a naturally-occurring phenomenon for expected utility maximisers.

How can a non-profit organization ensure that its future incarnations pursue the values of its founders? How do parents ensure that their children will pursue parental values? How do individuals ensure that when older they will pursue younger-version values?

Of course the degree to which anyone can do those things is limited, Robin, (unless someone uses an engineered superintelligence to do them), but do you seriously think the questions you pose have any relevance to how the creator of a superintellence can ensure the superintelligence will follow the intentions of the creator?

If your reply is yes, please say whether you have read Eliezer's Knowability of FAI.

Robin: How does a nation ensure that in several generations its government will pursue current values. How can a non-profit organization ensure that its future incarnations pursue the values of its founders? How do parents ensure that their children will pursue parental values? How do individuals ensure that when older they will pursue younger-version values?

Using, um, theorem-proving about cognitive programs executing on effectively deterministic hardware?

There clearly are limited degrees to which all of these things are possible, but it also seems pretty clear the degree is quite limited.

Sometimes I really do wonder if you and I are on the same page, and this is one of those times.

"Re: We win, because anything less would not be maximally wonderful.

Um, it depends. If we have AI, and they have AI and they have chosen a utility function closer to that which would be favoured by natural selection under such circumstances - then we might well lose.

Is spending the hundred million years gearing up for alien contact - to avoid being obliterated by it - 'maximally wonderful'? Probably not for any humans involved. "

Then wouldn't you rather we lose?

Tim: Although I don't implement CEV, I would hope that one of the first steps of an AI would be to move most everyone to living in a simulation and ditch our bodies, which would provide the sort of flexibility you're talking about without necessitating any radical changes of the human experience.

In other words, there's no substantive difference between the survival-oriented alien invaders and human-built survival machines, so why bother to pre-emptively institute the outcome of invasion?

I doubt the premises. That assumes a very powerful kind of convergent evolution is at work - and that both parties have fully converged. Neither of which seem very plausible to me. Chance will result in different forms - and one civilisation may have got closer to perfection than the other.

If a modestly increased risk of destruction by aliens is compensated for by much greater achievement of our aims, why should we instead abandon our aims in order to create exactly the outcome we supposedly fear?

I take it that your goals do not involve avoiding all your distant descendants being systematically obliterated. If you don't care about such an outcome, fine. I do happen to care.

AI creating human uploads

I'm sceptical. I think uploads are harder than many think. I think that the motivation to develop the technology will be lacking (once we have AI) - and that relatively few people will actually want to be uploaded. Uploading makes no economic sense: there are easier ways to produce simulated humans. Finally, it's not clear to me that there will be many people left by the time uploading technology is eventually developed - and those that do finally get a glimpse of cyberspace from within will need complete personality transplants - if they want to stand a hope of holding down any kind of job.

"I think uploads are harder than many think."

Maybe so, but with strong AI the problem would be quite simple.

"I think that the motivation to develop the technology will be lacking (once we have AI) - and that relatively few people will actually want to be uploaded."

I think that once brain-computer interfaces start to become common, the internet will start to host a number of common spaces for people to interact in that take heavy advantage of these interfaces. By uploading, a person gets a vastly improved experience in those spaces. If these environments are rich enough with BCIs to support telecommuting for most people, then real world interaction might start to take second place. In this scenario, uploading would seem to be quite attractive.

"I take it that your goals do not involve avoiding all your distant descendants being systematically obliterated. If you don't care about such an outcome, fine. I do happen to care."

In what sense is the descendant (through many iterations of redesign and construction) of an AI solely focused on survival, constructed by some *other* human my descendant or yours? What features does it possess that make you care about it more than the descendant of an AI constructed by aliens evolved near a distant star? If it's just a causal relation to your species, then what do you make of the following case: you create your survival machine, and it encounters a similar alien AI, whereupon the two merge, treating the activities of the merged entity as satisfying both of their 'survival' aims.

Where does your desire come from? Its achievement wouldn't advance the preservation of your genes (those would be destroyed), wouldn't seem to stem from the love of human children, wouldn't preserve your personality, etc.

Richard and Eliezer, I thought it obvious that strategies and degrees by which we now ensure our values are reflected in the future are relevant to the strategies we might later choose to achieve this same end in the future, and the degrees of success we might then expect. I will be surprised to hear persuasive arguments why this is not so.

Because the "strategies and degrees" by which we now ensure our values, reflect our helpless attempts to influence vast complex processes that we never created and whose components we can't specify, in essence by beating on them with a damp sponge. Compare human parenting to the art of programming an AI with control over each initial 1 and 0, carrying out a theorem-proof that the initial state has some property and preserves that property under self-modification? It just seems completely and obviously inappropriate, on the same level as Plato's Analogy of the Sun.

You might as well ask how a space shuttle could get to the Moon when no horse has previously succeeded in jumping more than a few meters. The modes of operation are incommensurable; pushing your leg on the ground is not like expelling rocket fuel downward. Likewise, beating on a human brain with the wet sponge of verbal suggestions, indeed anything you can do with the noisy wet human brain, is nothing like trying to program a deterministic CPU.

Eliezer, you have in mind a quite radical vision, of creating a provably friendly God to rule us all. For you even to be able to attempt this goal requires a scenario that satisfies some assumptions many of us doubt, and even if those assumptions go your way you admit it will still be extremely hard to achieve your plan. I thus think it quite reasonable to consider other possible scenarios and plans.

Eliezer: ...I'm seriously starting to wonder if some people just lack the reflective gear required to abstract over their background frameworks

I'm pretty sure of it, since I've seen some otherwise smart people make this kind of mistake (and I'm even more perplexed since I outgrew it right after my teenage years...)

In what sense is the descendant (through many iterations of redesign and construction) of an AI solely focused on survival, constructed by some *other* human my descendant or yours?

It is not realistic to think that one human can construct an AI. In the hypothetical case where someone else successfully did so, they might preserve some of my genes by preserving their own genes - e.g. if they were a relative of mine - or they may include my genes in an attempt to preserve some biodiversity.

Where does your desire come from?

I am the product of an evolutionary process. All my ancestors actively took steps to preserve their inheritance.

Its achievement wouldn't advance the preservation of your genes (those would be destroyed)

What - all of them? Are you sure? What makes you think that?

Maybe so, but with strong AI the problem [of making uploads] would be quite simple.

You mean to say that its solution would occur more rapidly? That is true, of course. It's the difference between taking mere decades, and being totally intractable (if it were a human project).

In this scenario, uploading would seem to be quite attractive.

To you, maybe. I reckon a poll to see if people would be prepared to have their physical brain destroyed in order to live a potentially-indefinite lifespan in a simulation, would not turn out very well. Show people a video of the brain-scanning process, and the results will be even worse.

> > Where does your desire come from?

> I am the product of an evolutionary process. All my ancestors actively took steps to preserve their inheritance.

Fallacy. Very few people have a *conscious intention* to preserve their genes. And yet, a very large number of people do end up preserving them, because their behavior patterns are designed by a process which reliably yields organisms that preserve their genes. We didn't evolve to be concerned about posterity, we evolved to have sex with each other, and to love and raise our babies (especially if we're female).

Tim,

What DNA are you loyal to? Mitochondrial genes? DNA inserted by a retrovirus in your father's sperm? Introns? Genes that you inherited with loss-of-function mutations so that they don't do anything? What if tonight, while you sleep, someone were to inject you with nanomachines that replaced your genome in each cell with a different one? Would you try to advance the interests of the new DNA or the old?

Where does your desire come from?

I am the product of an evolutionary process. All my ancestors actively took steps to preserve their inheritance.

Fallacy. Very few people have a *conscious intention* to preserve their genes.

Because a trait is rare it does not follow that it does not have a genetic basis. Besides, the basics of this trait are not especially rare: plenty of people want their descendants to flourish.

Of course, this trait - like every other aspect of me - is also the result of a complex developmental process, including interactions with the environment. That is just basic biology.

What DNA are you loyal to?

I do not always see myself as a single unified entity. Rather, I am a collective being - pulled in different, and sometimes conflicting directions by different genes - so: there is not necessarily any single unified "I" to which the "you" in your question refers. As with all organisms, my genes compete with each other to control my body. It's mostly a benign competition - often likened to a parliment. However, conflicts of interests can occur - in which case which gene gets the upper hand depends on the details of the circumstances.

How does a nation ensure that in several generations its government will pursue current values.

I'm reminded of the openining of Leviathan, which sounds like it's about AI, but is actually about government: "For what is the heart, but a spring; and the nerves, but so many strings; and the joints, but so many wheels, giving motion to the whole body, such as was intended by the Artificer? Art goes yet further, imitating that rational and most excellent work of Nature, man."

The things that can go wrong with an AI (behaving in accordance with the rules as written instead of the will of the designer) even resemble things that can go wrong in a legal system.

Does anyone know how, in the real world of today, one can modify one's brain to experience more pleasure? Current drugs either just plain don't work very well (Prozac, etc.) or result in a short-term increase in pleasure happiness at the cost of a long-term decrease in average pleasure (cocaine, heroin, etc.). Electric brain stimulation seems like it's less likely to lead to tolerance effects, but it's very difficult to find a surgeon willing to implant electrodes into one's brain or spinal cord because you want to wirehead yourself (and it doesn't seem to be as effective in humans as in rats, for whatever reason).

We win, because anything less would not be maximally wonderful.

What will you do if, having programmed an AI to be unable to act against its extrapolation of human preferences, it begins herding people into tiny pens in preparation of their slaughter?

Some of you seem to be making some rather large assumptions about what such an AI would actually do.

Re: Wireheading - and a short-term increase in pleasure happiness at the cost of a long-term decrease in average pleasure.

Heroin looks good to me - or at least is the best we have - unless a specific medical condition is making you sadder than normal.

You cannot reasonably expect to become a wirehead without your life going up the tubes - resulting in massive stimulation of your pain and unhappiness circuitry - but, by most accounts, heroin remains pretty effective at making all that go away - at least up until near the end. Behaviourally, a heroin addict is effectively indistinguishable from a wirehead.

As Renton said: "Take the best orgasm you ever had, multiply it by a thousand and you're still nowhere near it."

Who is it that's using these words "benefit", talking about "lesser" intelligences, invoking this mysterious property of "better"-ness? Is it a star, a mountain, an atom? Why no, it's a human!

...I'm seriously starting to wonder if some people just lack the reflective gear required to abstract over their background frameworks. All this talk of moral "danger" and things "better" than us, is the execution of a computation embodied in humans, nowhere else,

I don't find anything mysterious about the concept - and I certainly don't see that 'betterness' is embodied only in humans.

We need to presume that AIs are better than humans in at least one way - a way that is important - or your whole argument falls apart. If the AIs won't be better than humans in any way, what makes them more dangerous than humans are?

Electric brain stimulation seems like it's less likely to lead to tolerance effects, but it's very difficult to find a surgeon willing to implant electrodes into one's brain or spinal cord because you want to wirehead yourself (and it doesn't seem to be as effective in humans as in rats, for whatever reason).
Doug S., do you have any sources for that last claim? I've only ever come across one reference to a serious wireheading attempt in humans, and if I recall correctly it was a disaster - the human responded precisely as the rats did and had to have the wire's trigger taken away.

I suspect that any further attempts were intentionally directed so as not to be fully effective, to avoid similar outcomes. But new data could change my mind on that, if you possess some.

Caledonian:

... We ask Heath if human beings are as compulsive about pleasure as the rats of Old's laboratory that self-stimulated until they passed out. "No," he tells us. "People don't self-stimulate constantly -- as long as they're feeling good. Only when they're depressed does the stimulation trigger a big response. There are so many factors that play into a human being's pleasure response: your experience, your memory system, sensory cues..." he muses.

http://www.paradise-engineering.com/brain/index.htm

In the case of humans, our reward-pathways are (slightly) more anatomically diffuse than the average rodent. At least with present-day electrode-placement techniques, intra-cranial self-stimulation (ICSS) as practised by laboratory humans doesn't lead to uncontrolled hedonistic excess and death. Only depressed or deeply malaise-ridden human subjects compulsively self-stimulate when wired. Ill-defined "ethical constraints", however, are commonly held to forbid the human use of ICSS rather than to mandate its widespread adoption and refinement for "treatment-resistant" depression - even by avowed utilitarians. So instead of using depressed fellow humans, experimenters use rats. Pleasure-crazed rodents have become the symbolic expression of wirehead hedonism - and of all the pitfalls that "unnatural" pleasure entails.

http://wireheading.com/

Thanks, Doug S. I bet that first attempt I'd read of involved a depressed patient - although I wonder how many non-depressed patients it's actually been attempted on, given the highly invasive nature of the procedure.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31