« Setting The Stage | Main | Abstraction, Not Analogy »

November 19, 2008

Comments

Eliezer, I can't imagine you really think I disagree with anything important in the above description. I do think it more likely than not that life started before Earth, and so it may have been much less than nine billion years when nothing happened. But that detail hardly matters to the overall picture here.

"In the kingdom of blind chance, the myopic optimization process is king."
I sometimes chime in with pithy ways of summarizing arguments, so let me say: this sentence is a winner. I cannot top it.

Eliezer: Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already.

Why should you assume that they do not? Humans may not have detected other life forms, but you cannot deduce from that that they do not exist. They might not want us to detect them. It might not be possible for a civilisation to extend beyond its solar system.

"But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life"

And how do you suspect that is going to happen?

To add to Abigail's point:
Is there significant evidence that the critically low term in the Drake Equation isn't f_i (i.e. P(intelligence|life))? If natural selection on earth hadn't happened to produce an intelligent species, I would assign a rather low probability of any locally evolved life surviving the local sun going nova.
I don't see any reasonable way of even assigning a lower bound to f_i.

"Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already."

So do you thing that the vast majority of The Big Filter is concentrated on the creation of a first replicator? What's the justification for that?

I don't yet see why exactly Eliezer is dwelling on the origin of replicators. As Robin said, it would have been very surprising if Robin had disagreed with any of it.

I guess that Eliezer's main points were these: (1) The origin of life was an event where things changed abruptly in a way that wouldn't have been predicted by extrapolating from the previous 9 billion years. Moreover, (2) pretty much the entire mass of the universe, minus a small tidal pool, was basically irrelevant to how this abrupt change played out and continues to play out. That is, the rest of the universe only mattered in regards to its gross features. It was only in that tidal pool that the precise arrangement of molecules had and will have far-reaching causal implications for the fate of the universe.

Eliezer seems to want to argue that we should expect something like this when the singularity comes. His conclusion seems to be that it is futile to survey the universe as it is now to try to predict detailed features of the singularity. For, if the origin of life is any guide, practically all detailed features of the present universe will prove irrelevant. Their causal implications will be swept aside by the consequences of some localized event that is hidden in some obscure corner of the world, below our awareness. Since we know practically nothing about this event, our present models can't take it into account, so they are useless for predicting the details of its consequences. That, at any rate, is what I take his argument to be.

There seems to me to be a crucial problem with this line of attack on Robin's position. As Eliezer writes of the origin of life,

The first replicator was the first great break in History - the first Black Swan that would have been unimaginable by any surface analogy. No extrapolation of previous trends could have spotted it - you'd have had to dive down into causal modeling, in enough detail to visualize the unprecedented search.

Not that I'm saying I would have guessed, without benefit of hindsight - if somehow I'd been there as a disembodied and unreflective spirit, knowing only the previous universe as my guide - having no highfalutin' concepts of "intelligence" or "natural selection" because those things didn't exist in my environment, and I had no mental mirror in which to see myself - and indeed, who should have guessed it with short of godlike intelligence? When all the previous history of the universe contained no break in History that sharp? The replicator was the first Black Swan.

The difference with Robin's current position, if I understand it, is that he doesn't see our present situation as one in which such a momentous development is inconceivable. On the contrary, he conceives of it as happening through brain-emulation.

Eliezer seems to me to establish this much. If our present models did not predict an abrupt change on the order of the singularity, and if such a change nonetheless happens, then it will probably spring out of some very local event that wipes out the causal implications of all but the grossest features of the rest of the universe. However, Robin believes that our current models already predict a singularity-type event. If he's right (a big if!), then a crucial hypothesis of Eliezer's argument fails to obtain. The analogy with the origin of life that Eliezer makes in this post breaks down.

So the root of the difference between Eliezer and Robin seems to be this: Do our current models already give some significant probability to the singularity arising out of processes that we already know something about, e.g., the development of brain emulation? If so, then the origin of life was a crucially different situation, and we can't draw the lessons from it that Eliezer wants to.

Oops; I should have noted that I added emphasis to those quotes of Eliezer. Sorry.

I don't yet see why exactly Eliezer is dwelling on the origin of replicators.

Check with the title: if you are considering the possibility of a world takeover, it obviously pays to examine the previous historical genetic takeovers.

Oh, and it's called the Great Filter.

Robin, I didn't imagine you would disagree with my history, but I thought you might disagree with my interpretation or emphasis.

Tyrrell:
I think Eliezer is going to say that all previous trends were guided by processes that didn't involve self modifications of intelligence. That's the game-changer, whether it's hand coded or uploaded.

We know we are in for a dramatic finale when the history of the universe is recounted as prologue. Fortunate for us that the searchable neighborhood has always held superior possibilities. And one would expect the endpoint of intelligence to be when this ceases to be the case. Perhaps there will be a sign at the end of the universe that says 'You can't get there from here. Sorry.'.

Eliezer, as someone who as been married for 21 years, I know better than to try to pick fights about tone or emphasis when more direct and clear points of disagreement can be found. :)

to say nothing of a whole multicellular C. elegans earthworm

It would probably better if you had said nothing of it. It eats dead rotting vegetable matter, which you have just hypothetically removed. Plants are replicators too! So it would die in short order and time would be reversed.

But the significant thing was not how much material was recruited into the world of replication; the significant thing was the search, and the material just carried out that search.

Search is significant, but it was not the only significant thing. What was searched was also significant. If by chance a brain that searched the space of good chess strategies had spontaneously appeared, it would not be important. What was searched was "what patterns are good for survival to date", not "what patterns that are expected to be good for survival". This is important, it is real first hand information, we cannot exist because of some delusive part of natures mind that thinks we are good at surviving so far, we have to be!

Our existence is first order information about the world. Our mental models are only second order, they are one removed from reality. We try and update them by testing our reality against what the models predict, but there might always be black swans the models don't see. The machinery that creates them has to have been useful for surviving to date and carrying out many tasks that helps that survival, but the models themselves do not necessarily have to be useful, right, correct or true. I think there will always be a flow of information from the first order bodies to the second order models.

The issue, of course, is not whether AI is a game-changer. The issue is whether it will be a game-changer soon and suddenly. I have been looking forward to somebody explaining why this is likely, so I've got my popcorn popped and my box of wine in the fridge.

Tim Tyler,

I don't yet see why exactly Eliezer is dwelling on the origin of replicators.

Check with the title: if you are considering the possibility of a world takeover, it obviously pays to examine the previous historical genetic takeovers.

Right. I get the surface analogy. But it seems to break down when I look at its deeper structure.

I still don't get how Eliezer and (according to Eliezer) Robin think that life (or Life, whatever, a capital letter won't do the trick) is going to shape the stars and (if I understand correctly) the whole Universe. For all we know, despite all the replication and optimization, living things don't seem to have done much of that. Living things have shaped (to some extent) the surface of one planet. Big deal. If we changed the orbit of that same planet, I would think it impressive, compared to what we've done in the past. But still, big deal. It would only be one planet. Shape the Universe? How? And, if you don't know how, why do you think life will do it?

(Let me guess, intelligence can do anything. All right. Intelligence is God. But, why?)

"I still don't get how Eliezer and (according to Eliezer) Robin think that life (or Life, whatever, a capital letter won't do the trick) is going to shape the stars and (if I understand correctly) the whole Universe."

I don't get, how you can't get it. Really. It's a matter of a degree, how far away we make some changes. I trust, you wouldn't even if you could. But I would turn Jupiter and everything beyond to some useware, only if I could.

So "I" would remain, you would not. How difficult is to understand this?

- Thomas

I get what you're saying, Thomas, you're just answering another question. If some living thing were powerful enough to use a whole planet the size of Jupiter to its advantage, I agree with you that that would be the living thing that remains, rather than others. And then, if some living thing were even more powerful and could turn all the stars in the Universe into something useful, then that living thing would be the winner. But notice the "ifs". There's no if in "But Robin and I both suspect, I think, that the fate of the universe, and all those distant stars that outweigh us, will end up shaped by Life".

Felix: the power of our tools has no relation to our physical size or power. Stronger tools make stronger tools which make stronger tools still. You don't need to be the size of Jupiter to push a button, you don't need to be the size of a molecule to manipulate genetic code.

Yes, that's my point. We have traveled quite a long way, and there is no sign we couldn't go further with some luck. Producing useware from jupiters and stars is something one would expect. At least from a minority of civilizations, had been any of them around. But apparently, there is only a big void.

OTOH, they have still a lot of time to evolve to at least our current level from nothing, before we get there. That holds for the majority of the Universe. The Galaxy is another story, it's too small for this scenario. But the Universe as a whole is mostly more than 5 billion light years away. Our tomorrow Singularity could still be one of many, far apart. The SAI should ponder this immediately. What to do, not to be in war with others, coming from every space direction? The Ethics' space will be narrowed by this fact or guess.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31