« Gas Arbitrage | Main | Sustained Strong Recursion »

December 05, 2008

Comments

Robin: " Evolved Desires ...

...or it will be competitive, with conflicting powers each choosing how to perpetuate themselves. Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future."

Now, it seems obvious to me that most people do not want a world which is full of creatures who want to propagate themselves at all costs. We do not want to live in the crack of a future dystopia. When I say it seems obvious, I mean that no sophisticated mathematical analysis of maximally competitive scenarios is required to ascertain their ultimate undesirability.

Why do I claim that there is no need for such analysis? Because the underlying assumption for these "evolution of values under intense competition" scenarios is that you have to only care about perpetuating yourself, at absolutely all costs. Empirical fact: this is not a desirable state to be in, as far as the vast majority of people on this planet are concerned.

Let me define a limited competition scenario to be one where selection effects do not apply robustly. It seems to me that the only vaguely acceptable options are either a singleton or a fairly limited competition like the competition that people in developed nations with a welfare state find themselves in.

I think that the only really interesting question here is how stable a limited competition system can be, and how increasing technology affects this stability. Most people in the transhumanist sphere seem to be arguing that we need a benevolent singleton because limited competition scenarios are increasingly unstable as technology increases - they will lead to either an extinction event or a total competition scenario. I'd be interested to see this assumption challenged.

Roko, how can I challenge an assumption until arguments are offered in its favor? There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn't they? Treating a competition as a total war is only required when you expect others to treat it that way; give peace a chance.

"If total success over this time is the product of success factors for many short time intervals... A battle for the solar system is only one of many events where a lineage could go extinct in the next trillion years; why should evolved creatures treat it differently?"

What sort of factors are you thinking about for a singleton expanding into our limited and apparently uninhabited accessible region, with current physical limits (thermodynamics, no FTL, etc) assumed? Are you thinking about the entities' credence in the hypothesis that resources can increase vastly beyond those that physical limits seem to suggest? If resources could grow indefinitely, e.g. if there was a technological way to circumvent the laws of thermodynamics, then entities with unbounded utility functions (whether linear or log in reproductive success) will all have their calculations dominated by that possibility, and avoid struggles in the solar system that reduce their chances of getting access to such unbounded growth. I'm planning to talk more about that, but I started off with an assumption of common knowledge of current physics to illustrate dynamics.

"There cannot have been a history of previous last-extinction-events to select against creatures with preferences poorly adapted to such events."

Intelligent, foresightful entities with direct preferences for total reproductive success will mimic whatever local preferences would do best in a particular situation, so they won't be selected against, but in any case where the environment changes so that evolved local preferences are no longer optimal, those with direct preferences for total success will be able to adapt immediately, without mutation and selection.

Without wishing to put words in his mouth, Michael Anissimov seems to be arguing in favor of a singleton for exactly these reasons in his latest post:

http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/

He doesn't seem to consider strong competitive scenarios as a serious possibility, so he sees the future as a choice between a singleton and extinction. [Correct me if I'm wrong here, Michael]

Now, what arguments can I think of in favor of the instability of a limited competition scenario as technology increases? Well, one very general argument that has been aired many times is Eli's idea that we human beings are adaption executers, we execute a set of adaptions that used to maximize our inclusive genetic fitness in our EEA. Since the environment has changed a lot, we are actually *not* fitness maximizing creatures any more. This defeats the following argument you made:

There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn't they?

In order to get a qualitative idea of what a future full of genuinely fitness maximizing creatures will be like, we should look at natural environments where extant creatures genuinely maximize their genetic fitness. Nature red in tooth and claw.

**********************

When we talk about "giving peace a chance" in a *competitive* scenario that is *not a singleton*, we must be careful not to assume both a statement and it's negation. From Nick Bostrom's paper:

"All the possible singletons share one essential property: the ability to solve global coordination problems"

So, peace is possible in a non-singleton scenario if and only if it is in each agent's interest to peaceful. However, if we combine this with the basic axiom of a competitive scenario:

"Different agents have conflicting goals"

We get a global co-ordination problem that, by assumption, cannot be solved, and therefore it is not in each agent's interest to be peaceful.

The idea of a peaceful, competitive non-singleton scenario is self-contradictory.

Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future.

That seems effectively equivalent to a singleton, and an unfriendly one at that. (It does not care about us.) This is the big thing you think will shape the future, that almost no one will be able to get away from.

There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn't they?

Will we implement this understanding before or after hand held superweapons? I would really prefer a _better_ world for the next billion years, not just one with FASTER MORE POWERFUL monkeys fighting over stuff.

"Current formal modeling techniques are actually up to this task..."

This is an interesting research proposal, but the models used to investigate natural evolution make some assumptions that don't hold for scenarios about advanced civilizations. In particular:

1) Older generations do not automatically die off in some characteristic time. Instead they suffer a very low annual mortality that can be reduced to arbitrarily low levels by investing resources in survivability rather than reproduction.

2) Individuals can reproduce at any age, at a rate limited only by their available resources.

3) Rather than using resources to reproduce, many kinds of individuals can simply grow in size instead (i.e. an AI with thousands of bodies controlled by a single distributed consciousness).

4) Rather than having fixed preferences engineered by evolution, individuals can explicitly model the results of various strategies and switch from one to another as changing circumstances dictate.

It seems obvious that, while there would still be selection pressures in such a civilization, the traditional concepts of “species”, “individual” and “generation” don’t exist in anything like the same form. Building a model that can handle this situation does seem feasible, but it would be a big departure from the ones that currently exist.

Local Risk: Creatures should care about their lineage success, i.e., the total number of their gene's descendants, weighted perhaps by their quality and relatedness, but shouldn't otherwise care which creatures sharing their genes now produce those descendants. So they are quite tolerant of risks that are correlated, especially negatively, within their lineage. But they can care a lot more about risks that are correlated across such siblings. So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.
Huh?

What does "risks that are correlated, especially negatively, within their lineage" mean? And what is the connection between that, and global catastrophe?

Also, what does "logarithmic preferences over success now" mean? That preference is proportional to the logarithm of... what? Not "success now", surely, since we are contrasting short-term success with longer-term success. Again, what is the connection with global risks?

Technically, if U(prodtN rt) = sumtN u(rt)
Please explain what U, t, N, and u are.

How did you get subscripts and superscripts? HTML doesn't work.

Carl, you lost me. Your first quote of me isn't talking about a singleton, and I don't see how physics knowledge is relevant. On your response to your second quote of me, you can't just assume you know what sort of risk-aversion regarding the final outcome is the "true" preferences for "total success." If evolution selects for log preferences on all timescales on which it acts, why isn't log risk aversion the "true" total success risk-aversion?

Roko, just repeating the phrase "Nature red in tooth and claw" is just nowhere near a sufficient argument.

James, will you really argue detailed analysis is irrelevant because you just know that more powerful monkeys are always bad things?

Billy, yes some details change; that is why I'm calling for more analysis, so we can see how important those details end up being.

Phil, If there are two possible outcomes, 1 for me and 0 for you, or 0 for you and 1 for me, our payoffs are negatively correlated. The formula should define everything. My browser (Firefox) shows its sub-sups in the post, but not in your comment; no idea why.

I'll reply in a post.

I still don't see the connective tissue in your argument. A car accident and a global catastrophe are both correlated positively across siblings; how do your comments imply that they should be viewed differently?

"The formula should define everything."

The formula doesn't say what U, u, or r are. There is really and truly nothing in your post to indicate what U and u are. Some sorts of utilities, perhaps; but how do they differ? r is probably some measurement of risk, but I need more information, as it does not make sense to make utility a pure function of risk.

Robin: If evolution selects for log preferences on all timescales on which it acts, why isn't log risk aversion the "true" total success risk-aversion?

Entities with logarithmic preferences over their aggregate number of copies in total world-histories should behave sublogarithmically when making local, independent choices on the next generation. The evolutionary analysis similarly talks about entities that you are likely to see in the sense of their being most frequent, not entities whose logarithms you are likely to see.

You can't literally have logarithmic preferences at both global and local timescales, I think. If global preference is logarithmic, wouldn't local preference be log-log?

Anyway: would you agree that: a linear aggregate utility over complete world-histories corresponds to logarithmic choices over spatially global, temporally local options, whose outcome you believe to be uncorrelated to the outcome of similar choices in future times.

Robin:
They don't have to be always bad things, they are not reliably good things. Have we achieved friendliness yet?

Detailed analysis of potential competitive futures is great. I'm not saying this isn't important work. I AM saying detailed analysis isn't important to the question of if we need to do major work on friendliness. It may be VERY important to the question of how to obtain friendliness if Eliezer is wrong about the FOOM.

We know evolution isn't friendly.

Your argument seems to be that the bulk of the universe's resources will go to entities who are best at snatching them up (either for themselves or for their descendants or kin), and we can tell certain things about those entities attitudes towards resources and kin and risk.

You seem to be arguing that the universe will be dominated my beings that want to take as much of the universe as they can (or their descendants wanted it for them.) and care mostly about those most closely related to them.

These are not compatible with my values. ESPECIALLY if I'm not closely related to those entities, and much less powerful.

My point is that YOU are arguing that the future will be filled with super entities entities that would be unfriendly to me, and you, and I think the bulk of humanity (both now as it is now, and as it would be then). Our only advantage against them is that we are in their past.

So we need to either to work on a friendly singleton if a singleton is feasible, work on imposing further selection effects to encourage friendliness (using evolution as a singleton) or somehow ensure that all the initial competitors will be friendly, so that there are no unfriendly lineages.

What past trends point to a peaceful future at the hands of evolution and super-monkeys?

(Actually, The entities described seem more like von Neumann probes with extra personality. What will entities evolved to be colonizers do when the universe is full? Why should I value that future?)


Eliezer, I think you are just mistaken; log preferences aggregate or split in time to log preferences. Regarding your last question, I said a wide range of preferences over final outcomes, including linear preferences, converge to log preferences over each step.

Phil U(x) is your final preferences over resources/copies of x at the "end." r_t is the ratio by which your resources/copies increase in each timestep. u(r_t) is your preferences over the next timestep, expressed in a linear form so that if probabilities and choices are independent across timesteps, then in order to maximize U, you'd just make your choices about r_t maximize the expected value of u(r_t).

Eliezer, I think you are just mistaken; log preferences aggregate or split in time to log preferences.

Ah, okay, I see my problem. I was assuming that taking the log of population sizes just put us into a log-world, exchanging multiplication for addition. But in the new world, options add fixed amounts to your current total, regardless of your initial position, so preferences are just aggregative (not logarithmic) in the new world.

(Thinks.)

I think what this reveals is that for repeatable choices with a certain kind of temporal independence and an indefinite time horizon, your local preferences will start corresponding to a representation under which the effect of those choices is purely aggregative, if such a representation exists. A representation where -4 units of negative is exactly balanced by +1 and +3 positive outcomes. As your time horizon approaches the indefinite, such an approach will dominate.

If you expect to encounter lots of options with non-multiplicative effects - like "this will square my population, this will take the square root of my population" - then you'll be wise to regard those as +1 and -1 respectively, even though a logarithmic analysis will call this +X vs. -0.5X.

My point is that YOU are arguing that the future will be filled with super entities entities that would be unfriendly to me, and you, and I think the bulk of humanity (both now as it is now, and as it would be then). Our only advantage against them is that we are in their past.

So we need to [snip plan]

It sounds as though you want to block the natural evolutionary process - turning yourself into a parasite on the glorious beings of the future, for your own benefit. Enslaving our offspring may be possible - but do we really want to screw over our own legacy in that way? What will "human friendliness" do for civilization's spaceworthyness? How long do you think you would be prepared to keep development bogged down in the human phase for?

Eliezer, it sounds like you are probably right with your ending comment, though it could be interesting to hear it elaborated, for a wider audience.

It's interesting that this aspect of the debate comes down to whether people's utility functions are (approximately) logarithmic, and equivalently whether they are risk averse or risk neutral. The paper Robin cites seems to show that evolution should gift us with logarithmic utilities, making us risk averse. And indeed the psychological literature largely confirms this. The basic argument in the paper is that the overriding consideration built into us from evolution is avoiding extinction. As the author notes: "survival must have been a dominant ability, because among the trillions of predecessors of a person living today, from the first cell to his own parents, there wasn’t a single one that died before reproduction age." We are built to avoid destruction rather than to maximize success.

Billy Brown asks whether we can't just override this, for example deciding that risk neutrality is better. On one hand it sounds almost incoherent to propose changing our preferences, since by definition our preferences are what we try to satisfy. At the same time, it does seem that we perceive a contradiction between our desires for success and the timidity which risk aversion imposes on us. Maybe this is an example of evolution screwing up, as it did with sex, and using too many layers of indirection in imposing the desired behavior. Rather than making us want to reproduce, it makes sex feel good; but we can separate the two. In the same way, rather than making us value log(success) directly, it makes us want success but afraid when we take risks; yet we can find ways to overcome the fear.

In the end though I wonder if it is true that the shape of this utility function will decide and determine whether the future will include an attempt to grab for all resources available, wiping out all competitors. Are risk neutral entities fated to wage all out war even when the probability of success is low, due to the great rewards which might be gained? Are risk averse entities automatically going to be more interested in trade and mutual cooperation by virtue of their logarithmic utility functions? It seems odd for this fundamental fact about the future to turn on such a specific and seemingly esoteric technical matter.

Roko: "We get a global co-ordination problem that, by assumption, cannot be solved [...] The idea of a peaceful, competitive non-singleton scenario is self-contradictory."

I'm pretty sure this is begging the question. Why assume the coördination problem can't be solved? If none of the competing entities can overpower the others, then they may well find it beneficial to trade or coöperate peacefully. Sure, each might prefer to take over entirely, but when that option isn't on the table, an eternal stalemate (peace) could be the next best thing. Of course, said peace could still end up destroying everything we value.

I don't see how the "adaptation-executers, not fitness maximizers" point is relevant (and for the record, the citation should really go to Cosmides and Tooby, not Eliezer). I think Robin's point is that humans do compete (e.g., in markets) without slaughtering each other; that we're competing for other things than fitness doesn't really change the argument.

Tim Tyler: "[B]ut do we really want to screw over our own legacy in that way?"

Yes. You seem to be suggesting that a future filled with "Nietzschean" fitness-maximizers racing to burn the cosmic commons would be "glorious"--but why? That scenario has absolutely nothing to do with sorting pebbles into correct heaps--which, as we all know, is the only thing that matters.

Tim:
This is as much about them as it is us. Robin portrays a world in which ems may be routinely deleted because they are no longer economical. Enslavement and murder of what are essentially humans is an outcome I want to prevent. Giving the universe to whatever the most brutally efficient uploads are is just the undesirable situation most directly suggested by this post.

Not murdering EMs to make more EM's is not enslaving our offspring. I don't think making AI friendly as opposed to unfriendly is either. In any case, DEVELOPMENT doesn't have to be bogged down much at all. Our current resource needs are small, compared to what is available

I'm not precluding that a Friendly AI will forcibly upload and enhance everyone, either. I'd prefer people have options but a FAI might know better.

Even a war between morally opposed sides could be resolved by cooperating in one-off prisoner's dilemma, with both sides choosing to avoid the war that isn't expected to be reliably won by either side, incorporating the eternal truce in their morality, so that they won't need to worry about either hot or cold war in the future, and will just go about their business like each of them is alone in the universe.

Tim: (I don't know if I should belabor this point, but I don't think anyone's brought up yet the fact that) a human-Friendly AI could rapidly colonize the universe with pure AI probes (with stable goal systems and defenses against mutation, to prevent commons-burning), even if all humans stayed biological (which is, of course, a ridiculous distortion of Friendliness).

"Selection effects apply robustly to competition scenarios"

This depends on the number of entities competing and the strength of selection. At selection coefficients approximately equivalent to the inverse of the effective population size, chance events will dominate the dynamics.

So they are quite tolerant of risks that are correlated, especially negatively, within their lineage.

I find that statement quite confusing. I can see two ways out: one is that "lineage" means descendants and excludes siblings.

The other is that it is some kind of typo. I would change it to: "they are quite tolerant of risks that are not correlated, or negatively correlated; but they are sensitive to risks that are positively correlated across siblings."

Also, Tim:
Any arguments about what we want to do to our offspring is appealing to the exact same morality that the Friendly AI will have. Yes there could be situation where the AI would kill many people to save many more, if our morality accepts that.
The AI will also weigh to good of the spaceworthyness of civilization against other conflicting interests. Those conflicting interests will be other kinds of things WE would care about. Whatever its decisions, it will have good reasons that both of us would likely agree with, if we understood the particular situation.

Building an AI that is not friendly is essentially acting as if you don't care about something you care about.

There is a bit of rhetorical judo here. As it is defined, you can't make any compelling argument that a FAI would behave in any way we would view as a downside. As soon as I view a course of action as not the best, it also becomes what I would predict the FAI NOT to do.

I'm not sure if this represents a problem with how FAI is defined.

Robin: I usually criticize economics for its assumption of non-myopic agents and here I see you casually discarding that assumption and assuming myopic agents when it supports you in an argument in a situation which involves non-humans who should be able to self-modify to actually be non-myopic. Does this really constitute a good faith effort to give the argument that seems best justified by the evidence rather than to argue for a position? Everyone agrees that the adversarial process doesn't result in agreement among disputants, so taking a side in this manner seems contrary to the supposed purpose of this experiment.

Hal: It seems to me that the future of the universe hinging tightly on what casually seem like technical questions is exactly what we should expect a-priori given that our casual intuitions aren't very reliable outside of their domain or origin.

Hal, if we can understand whatever the future does depend on, why shouldn't math models help that understanding?

Josh, yes, agreed.

Douglas, yes that was a typo; fixed.

Michael, I'm not assuming myopia.

Re: glorious competitive future: I don't know whether self-directed evolution is either likely or desirable.

Re: immoral future: I tend to think that the shifting moral zeitgeist means that our descendants will be more moral than us. There will probably always be dead wood that needs discarding. Death seems bad today because it takes with it a library of memories - but in the future, memories will be able to be rescued - if they are deemed worthwhile.

Re: probes that can't evolve: their genes would have to be protected - and it would be necessary to ensure that they could not spawn new cultural replicators - lest a new kind of evolution take hold and usurp that of its creators. Crippleware probes would be better than allowing evoulution on other planets to continue without monitoring - but worse then sending proper creatures.

Having superintelligence wipe us out is only one way to go wrong. Building a whole bunch of things about humans into the utility function of an AI seems like an obvious mistake to me - but perhaps it needs pointing out that attempts to build in dependencies on legacy systems can subsequently go on to cause problems.

Z.M. davis: "I'm pretty sure this is begging the question. Why assume the coördination problem can't be solved? If none of the competing entities can overpower the others, then they may well find it beneficial to trade or coöperate peacefully. Sure, each might prefer to take over entirely, but when that option isn't on the table, an eternal stalemate (peace) could be the next best thing. Of course, said peace could still end up destroying everything we value."

Well, indeed. Consider a world populated by a set of agents who all spend all of their available time and energy on elaborate defenses to prevent other agents from killing them, and suppose further that these defenses are almost universally effective. [So we are in a world where defense is much stronger than offense]. It must be the case that if a particular agent deviates from the strategy of spending almost all of his time and energy on defense, he will quickly be killed by other agents: this is the assumption of evolved desires through competition.

This could all be the case without any global coordinator like a world state, so we are not in the singleton scenario. But one has to ask: does this scenario really count as peace? Does it count as a solved coordination problem? (my answer: no) I certainly agree with you that it doesn't count as good(!), which means that we agree on the relevant point: Robin is busy analyzing the grim details of scenarios that we don't want to happen.

Vladimir Nesov: "with both sides choosing to avoid the war that isn't expected to be reliably won by either side, incorporating the eternal truce in their morality, so that they won't need to worry about either hot or cold war in the future, and will just go about their business like each of them is alone in the universe."

If those two sides constitute all of the agents in the world, such an "eternal truce" obtained by rewriting each others' moralities is another way of saying "singleton".

James Andrix: Your argument seems to be that the bulk of the universe's resources will go to entities who are best at snatching them up (either for themselves or for their descendants or kin), and we can tell certain things about those entities attitudes towards resources and kin and risk. You seem to be arguing that the universe will be dominated my beings that want to take as much of the universe as they can (or their descendants wanted it for them.) and care mostly about those most closely related to them. These are not compatible with my values...

- Exactly. I agree completely. If this is obvious to me, James and (I think ) to Z.M., why is it not obvious to everyone? Perhaps I am making some huge error here, which I'd be obliged if someone could point out. Perhaps I should spell out my argument again:

1. Evolved desires assumption <==> the vast majority of future agents will care only about propagating themselves, to the exclusion of all else.

2. Any world satisfying this assumption is not a world I want to live in, because I do not consider such beings to be value-bearing entities.

I gotta say it is kinda creepy how many commentators here at OB are so eager to build a God to rule us all that they cannot imagine wanting any other world, instead of because some tech problem forces us to choose between making a God and extinction. Does feel like cult-land.

@Roko:
Probably, efficient world order is always in a singleton, with all "local" activity optimized with respect to the rest of the world. For example, Friendly singleton needs to initially portion the moral pie among slightly different individual humans, the situation is similar. "Singleton" is not so much one of the many possible ways of building the future, but a way to characterize optimal solution in general.

Robin: "they cannot imagine wanting any other world,"

I can imagine wanting other kinds of worlds. I'd be quite happy with some of the following non-singleton worlds:

1. Worlds in which agents have conflicting desires, but are non-rational in a certain kind of innocent way: they simply don't realize that they can satisfy their desires by screwing other people over. There is some legend about an isolated community of pacific islanders who did not realize that it was possible to lie until western civilization arrived. Global co-ordination problems would never arise here.

2. Worlds in which agents can have conflicting desires, but by some lucky coincidence they never do. Global co-ordination problems would never arise here.

3. Worlds in which global co-ordination problems never escalate beyond the trivial level for some odd reason. For example in this world agents argue about whether to have meat or fish for dinner, but they don't have wars or kill each other. These kinds of worlds would (ethically) be like living in a children's book, everything would always turn out alright in the end. Clearly there is a whole continuum here - there exist worlds in which there is some medium level of conflict, perhaps involving shouting, fights, lying, defamation, mild hatred, etc but no killing.

Note also that the term "singleton" is somewhat vague but very general. Equating "singleton" with "god" is unfair IMO. I quote nick:

A democratic world republic could be a kind of singleton, as could a world dictatorship. A friendly superintelligent machine could be another kind of singleton, assuming it was powerful enough that no other entity could threaten its existence or thwart its plans. A “transcending upload” that achieves world domination would be another example. Yet another way in which a singleton could form is through convergent evolution, e.g. if it turns out that all sufficiently advanced individuals or cultures come to accept fundamentally the same values or goals. These common values in combination with all the individuals and cultures that embrace them would then be an “agency” in the broad sense intended here, and it would constitute a singleton. One could also imagine a singleton arising from the universal spread of a single self-enforcing moral code. ... A singleton need not be monolithic. It could contain within itself an enormous variety of independent agents each pursuing their own disparate goals, just as is the case in a liberal democratic state. The goals and actions of the singleton could be decided by its inhabitants or their elected representatives. ... The concept of a singleton is thus much broader and more abstract than concept of a world government. A world government (in the ordinary sense of the word) is only one type of singleton among many.

Nevertheless, all the possible singletons share one essential property: the ability to solve global coordination problems.

@Robin: Well, either you and I have really different visualizations of what the coherent parts of humanity's reflective equilibria would look like, or you don't think the Friendly AI project has the described outcome, or you have a really different moral reaction to that outcome.

If an AI goes FOOM, you seem to recognize that condition, or that prospect, as "total war". Afterward, you seem to recognize the resultant as a "God", and its relation to humanity as "rule". So either we've got really different visualizations of this process, or we have really different moral reactions to it. This seems worth exploring, because I suspect that it accounts for a large fraction of the real fuel in the argument.

Tim:
A Mature FAI could modify its own utility function, if it wanted to. If it had good reasons that you and I would find compelling. There is no lock-in other than doing what we would want and not what we wouldn't want.

Robin:
I can naively imagine many post singularity scenarios with no FAI that I would want to live in. I didn't understand that the many of the high probability scenarios were so awful until you explained it to me.

Extinction is just one of the awful things the singularity could bring.

I'm all for uploading, nanotech, and and agressive colonization wave, but what's the benefit of it all? I previously view these things as potentially dangerous but overall good things to have around. Now I come across an argument that the dangers I supposed are the most likely primary effects!

This SEEMS to me like values argument, that you are ok with the described slavery and murder. You've disclaimed endorsement of the future you describe, and I'm giving you credit for not being a murderous slaver... But you also don't seem to want to do anything to alter this future.

A strategy for changing this future doesn't need to take the form of a singleton. It CAN'T rely on a singleton, because part of the argument is that singletons aren't likely. God is not an option, so we have to use evolution. (or somethng else)

I think it's a fairly common goal among singularitarians that no sentient being will have to do anything they don't want to do. We enjoy life, and the nanites do all the work. That is what I see as the benefit to these technologies, that is what they are FOR. To serve us, and our goals. Going from that to a future where people get deleted for being uneconomical is losing a LOT. Getting a better future is well worth a slightly delayed colonization wave.

If you assume perfectly rational agents, then they will optimize over the whole future and it will be as if there is only one round of selection, so it will select for agents trying to maximize expected population. Such agents will recognize the main bottleneck and be risk-neutral about it. At some point, we will enter that regime, unless there is insufficient diversity to allow that outcome; a founder effect is much like a singleton.

If you assume perfectly rational agents, then they will optimize over the whole future and it will be as if there is only one round of selection, so it will select for agents trying to maximize expected population. Such agents will recognize the main bottleneck and be risk-neutral about it.

What state a superintelligent agent will aim at - and its level of risk aversion - will be properties of its utility function. You can't easily generalise over all such agents - because they could be very different from one another.

Douglas, as I told Carl, you can't just assume you know that maximizing expected population is "perfectly rational" choice. I've argued evolution selects for maxing expected log (quality-weighted) population (when that is the only long term resource).

Robin: You CAN assume that you will find the largest populations are the populations of those rational agents that maximize population, or roughly equivalently, that most resources are held by those rational agents which maximize resources under their control. Non-myopic agents don't need to let evolution sculpt their utility functions. They can adjust their behavior to evolutionary realities simply by understanding evolution

Seriously Robin, just think about the situation described very carefully. It's a math problem. There's only one right answer. Douglas Knight, Carl, Eliezer and I are all very reliable sources for correct answers on published math problems, including answers on whether the word problem translates into the math you are claiming it does.

Michael, Above, Eliezer seems to accept my math claim. I don't yet know if Carl is convinced or not. My disagreement with Douglas was about what "rational" means, which is not a math question. I certainly agree that all else equal you can expect to see more X in creatures that max X, vs. creatures that max log(X), but that is a different question from what preferences evolution selects for.

Yes, but it's the question we need to ask in order to understand what the hardscrapple frontier would probably look like.

I don't consider myself a superreliable math source. If the fate of the world isn't at stake, I'll often state an intuition rather than trying to prove it. For that matter, if the fate of the world were at stake, the first thing I'd do would be consult Marcello.

Robin, I accept the part about locally logarithmic behavior on spatially global and temporally local problems when there will be many future options and all are multiplicative. I don't accept the claim that evolution turns future entities into log-population maximizers. In a sense, you've actually shown just the opposite; because aggregative maximizers or log-maximizers will both show instrumental log-seeking behavior, entities with terminal log valuations have no fitness advantage. Evolution requires visible differences of behavior on which to operate.

If there are many non-multiplicative options - say, there are ways to form trustworthy contracts, and a small party can contract with an intergalactic Warren Buffett "I will give you 10% of my lineage's resources now, if you agree to use the same amount of resources to recreate copies of me in a billion years," then it's not clear to me that logarithmics have an advantage; most of the numbers might be in aggregators because numbers are what they want, and that's what they use non-multiplicative options to get.

Eliezer, I agree one might analyze non-multiplicative worlds, but no one has done so yet, and the world so far has been pretty multiplicative. Please recall that I was initially responding to confident claims by Carl and others that evolution would make for terrible wars over the solar system because evolved creatures would be terminal-outcome-oriented and risk-neutral about such outcomes. In this context I make three claims:

  1. It is not obvious evolution would create terminal-outcome-oriented creatures,
  2. It is not obvious such creatures would be risk-neutral about terminal outcomes, and
  3. Even if they were they would have to be rather confident this conflict was in fact the last such conflict to be risk-neutral about resources gained from it.
Do you disagree with any of these claims?

I used "rational" to mean not myopic. You can just assume incorrect usage, but you can't make me like it.

Evolution of myopic agents often produces ones that maximize log-resources. Evolution of non-myopic agents produces ones that maximize resources.

The question is: will there be enough non-myopia and diversity among Ems that the second effect kicks in? I expect evolution of Ems to head towards non-myopia, so the question is only about diversity, after the escape from myopia.

(By "evolution" I mean sampling afterwards by resource weighting.)

Douglas, you seem to just ignore the math I'm presenting and citing; do you know of some proof that makes it just completely irrelevant? Do you know of any formal analysis of the selection of non-myopic agents that supports your claim?

The work you cite assumes myopia. You seem to just ignore that.

The non-myopic case is very simple, at least assuming zero-sum resources (by which I mean the thing that we use to weight the final observation), which I should have said. The key point is "they will optimize over the whole future and it will be as if there is only one round of selection," or, as other people put it, only terminal values matter. To analyze the case of one zero-sum round of selection, there is nothing more to say than "linearity of expectation."

I don't know about evolution creating terminal-outcome-oriented creatures, but the case for self-modifying AIs by default converging to expected utility maximization has been written up by e.g. Omohundro. But I think that what you mean here is aggregate valuation by expected utility maximizers. This wouldn't be created per se by either evolution or self-modification, but it also seems fairly likely to emerge as an idiom among utility functions not strictly specified. Other possible minds could be satisficers, and these would be less of a threat in a competitive situation (they would only take over the world if they knew they could win, or if they expected a strong threat to their button-to-keep-pressed if they weren't in sole charge of the galaxy).

I'm frustrated that I seem unable communicate what should be a precise technical claim: evolution need not select for creatures who maximize expected future descendants. People keep claiming this as if it had been proven, but it has not, because it is not so.

The paper I cite is a clear precise counter-example. It considers a case where choices and probabilities are independent across time periods, and in this case its is optimal, non-myopically, to make choices locally in time to max the expected log of period payoffs.

That case easily generalizes to chunks of N periods that are correlated arbitrarily internally, but independent across chunks. Again agents max the expected sum of log period returns, which is the same as maxing the expected sum of chunk returns. And you can make N as large as you like.

The simplest example demonstrating the same thing is the environment in which actions have no consequences. In that environment, there is no differential selection between expectation maximizers and expected log maximizers. cf Eliezer

Robin:
I think part of the problem may be confusion of future descendants with future control of resources.

Robin-The paper I cite is a clear precise counter-example. It considers a case where choices and probabilities are independent across time periods, and in this case its is optimal, non-myopically, to make choices locally in time to max the expected log of period payoffs.

Douglas-Evolution of non-myopic agents produces ones that maximize resources.

What I get from Douglas is that every 'generation'of an EM or AI can plan ahead the whole future and behave optimally (for resources) in each situation. (or create offspring that behave optimally)

What I get from you is that creatures that have evolved logarithmic preferences do the best, in resources.

I think he's trying to answer the question:
"Even if you somehow knew that it was in fact that last extinction possibility forevermore, how could evolutionary selection have favored a different attitude toward such that event?"
with 'By making things smart enough to do the math with a linear utility function.'

And I think you're answering: 'They can but that's a losing option.'

You seem to agree that Evolution will create resource grabbers. It's tautological that the universe will go to those who are best at getting the universe, whatever that means. (But I bet it's not meekness)

Non-myopic agents can indeed ration their resources and avoid short term reproduction for the sake of their own long term success. I'm not sure if this is what you are asking for - but it seems like a pretty trivial observation to me: they form a government, and the government imposes rationing. We have seen plenty of historical examples of such rationing.

Reproductive success is indeed not evolution's utility function - see the work of Dewar - or my essay: God's utility function.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31