« Trust Us! | Main | Transparent Characters »

October 26, 2008


I agree, generally, but I've got a quibble with your last paragraph. I'm tentatively in agreement that stars are not, qua stars, optimization processes, but I'm less certain that stars do not contain optimization processes. And I'm tentatively certain that stars are the products of strong optimization pressures; how likely is star formation? Doesn't cosmological/astronomical evolution (i.e. the 'rules' by which it occurs) count as a (powerful) form of selection? There are innumerable dust clouds in the visible universe that never became stars.

Have you read 'A New Kind of Science' (yes, it's a pretentious title) by Stephen Wolfram? He has a number of interesting discussions of intelligence. Your recent posts re: intelligence and optimization processes remind me of Wolfram's statement (I'm greatly paraphrasing) that a sufficiently general definition of intelligence (in terms of information processing and something akin to your optimization processes) would necessarily include all kinds of entities that we would not categorize as intentional.

Tromp gives 2.081681994 * 10^170.

How do you avoid conflating intelligence with power? (Or do you, in fact, think that the two are best regarded as different facets of the same thing?) I'd have more ability to steer reality into regions I like if I were cleverer -- but also if I were dramatically richer or better-connected.

Re intelligence vs. power: think about it as quality of engine vs. how you apply the engine. You can have a perfect car, and still drown it in a pool. Quality of engine is abstract knowledge about it, it's not derived from performance of that engine on a specific race track. If you are highly intelligent, but get hit by a stone fallen from the sky, it doesn't mean that it was your preferred outcome. When the outcome turns out wrong each time, it shows that your abstract knowledge was wrong, but quality of engine is not outcome itself.


Kenny, compared to cumulative processes like natural selection that mutate and select over and over - let alone human intelligence that navigates a compressed search space - to select one star cloud out of a million possibles is only 20 bits of information at most. And considering the variance between star clouds, that 20 bits of information won't go very far. You can't expect to find complex functionally optimized machinery within stars on the order of what exists in the smallest bacterium. If evolution has not gotten started anywhere else in the galaxy, then I fully expect that the rest of the entire Milky Way contains less interesting complexity than one Earth butterfly.

See e.g. "No Evolutions for Corporations or Nanodevices".

Vladimir, if I understand both you and Eliezer correctly you're saying that Eliezer is saying not "intelligence is reality-steering ability" but "intelligence is reality-steering ability modulo available resources". That makes good sense, but that definition is only usable in so far as you have some separate way of estimating an agent's available resources, and comparing the utility of what might be very different sets of available resources. (Compare a nascent superintelligent AI, with no ability to influence the world directly other than by communicating with people, with someone carrying a whole lot of powerful weapons. Who has the better available resources? Depends on context -- and on the intelligence of the two.) Eliezer, I think, is proposing a way of evaluating the "intelligence" of an agent about which we know very little, including (perhaps) very little about what resources it has.

Put differently: I think Eliezer's given a definition of "intelligence" that could equally be given as a definition of "power", and I suspect that in practice using it to evaluate intelligence involves applying some *other* notion of what counts as intelligence and what counts as something else. (E.g., we've already decided that how much money you have, or how many nuclear warheads you have at your command, don't count as "intelligence".)

what about chaotic neutral?

This is very similar to an earlier post. Eliezer, go faster. I, for one, am waiting for some non-trivial FAI math - is there any?

Stars are optimisation processes. They maximise entropy production. See the work of R. C. Dewar. Dewar's work was based on the work of E. T. Jaynes in this area.

Vladimir, there's plenty of non-trivial decision theory math that other people have invented.

If I were going to write up a piece of novel FAI math, it would be my general theory of Newcomblike problems, but the last time I tried that it started turning into a book. But that would be the main thing that I would do, if it got to the point that impressing people with at least one piece of elegant math turned into a high enough priority.

The state spaces of both chess and go have to be expanded to include "previous history". In the case of go, this is to avoid the ko problem -- moves that would produce the exact same board state as a previous state are forbidden. Chess, well, see a chess rule book, or wikipedia.

When you drive to the supermarket, you aren't really aiming for the supermarket, you're aiming for a region of possible futures in which you don't starve.

The first time you head to the supermarket it might be trying to stop starving, but after you have done it for a few times, it becomes an in grained habit. And you might be heading to the super market, because you always head to the super market on a saturday. You might do so even if it becomes knowably sub optimal for starvation prevention, because you aren't re-evaluating the economics of doing so.

This is a known and exploited facet of humans, hence special offers to get you buying food for a bit, to hope it becomes a habit.

I however sometimes spend far too long making food buying decisions (i.e. more time spent if I was working for that time than I might save).

If your AI was making a decision to buy bread vs making it would it calculate the nutrition of store bought bread, cost of petrol and bread, wear and tear on the car etc vs the time taken to make bread, energy costs of cooking it, ingredient costs, wear and tear on oven, cost to repair or replace oven etc every time one of these values changed or would it use cached results?

Of course if the AI was going to be truly powerful it would be able to predict the secondary effects of choices it made, would the money it spent in avoiding starvation be in the hands of other people trying to avoid starvation of people in general, i.e. giving it to people spending it to encourage research and actions that lead to the avoidance of food shortages in the future (either through lobbying government or directly depending what was more influential). I sometimes feel I need a very Powerful AGI just to buy my groceries.

I think it's worth mentioning that Kasparov will have a harder time accurately predicting your moves than you will have predicting his. Each of you knows that Kasparov will win, but this will much more likely be due to a blunder on your part than a brilliancy on his. He may well reason, "sooner or later this patzer is going to hang a piece", but he will have no way of knowing when.

You cannot even drive to the supermarket without planning - it will take you a long, long time to arrive if you make random turns at each intersection.

Eliezer, what do you mean by "planning"? The word needs a technical definition. (So does "optimisation", for that matter, by people on both sides of the claim that stars are, or are not, optimisers.)

Reaching goals does not necessarily involve planning. Water reaches the foot of a hill without planning. The room thermostat maintains the temperature without planning. It is said that no plan of battle survives contact with the enemy. I have a simulation of a robot that walks over uneven terrain and hunts food particles. That robot does no planning. I know: I created it. (Neither does it optimise, learn, adapt, remember, or predict. It merely works.) I know my way around the town I live in, and do not need to make any plan to reach the supermarket. I only need to know what to do at each point of the journey. In a strange town I would use a map and make a plan, but the plan would have to be continually updated according to conditions encountered.

To leap from observing the accomplishment of goals to "planning" is an anthropomorphisation of the sort that you have condemned in people working on AI, except when planning, technically defined, is demonstrated to actually be present in the mechanism. If "planning" does not mean some particular sort of mechanism, then you might as well call it "emergence", or "pixies".

you try to steer reality into a particular region of possible futures.

By "steering" I understand continually correcting your course to maintain the approach to your goal. Steering is not planning. The thermostat steers the temperature and the robot steers to its prey without planning. Steering without planning is possible; planning without steering is useless. Relying on a plan makes it unreliable.

Anyone can test this. Give someone directions in the form of a plan of which way to go at each intersection. Don't tell them the destination you're aiming them at. Have them execute the plan and tell you where they got to.

Will, you still seem to think Eliezer is saying "do what an optimal agent would, even when doing so is not optimal." No.

[Richard] Anyone can test this. Give someone directions in the form of a plan of which way to go at each intersection. Don't tell them the destination you're aiming them at. Have them execute the plan and tell you where they got to.

But then they're not optimizing for a goal they can be said to have. If a street is closed, they won't reach the destination, as they would if they knew what the destination was.

+1 to Will Pearson and Richard Kennaway. Humans mostly follow habit instead of optimizing.

Eliezer, this is interesting:

> my general theory of Newcomblike problems

Some kind of bounded rationality? Could you give us a taste?

Vladimir: Some kind of bounded rationality? Could you give us a taste?
I was wondering the same thing. I'd be less interested in a decision theory for sub-Omegas and more interested in one for Omegas.

Tim Tyler: Stars are optimisation processes. They maximise entropy production.

No, at least, not by the definition of "optimisation processes" that Eliezer uses. Stars don't plan to maximize entropy productions, gravity doesn't plot a course down a hill so that a ball reaches the lowest point. Both can get stuck in a local maxima. Reread the post about Kasparov, and try to come up with a corresponding story about stars. Elizier is not talking about the same kind of thing as you are.

No, at least, not by the definition of "optimisation processes" that Eliezer uses.

You are being ridiculous - "optimisation process" is a perfectly standard term.

Stars don't plan to maximize entropy productions.

"Planning" is not part of the definition of what constitutes an optimisation process.

an outcome that ranks high in your preference ordering

Well if Garry's wins are in the centre of your preference ordering circle of course you'll lose! Some fighting spirit please!

Oh, and if something maximising entropy is a valid optimisation process, then surely everything is an optimisation process and the term becomes useless? Optimisation processes lead (locally) away from maximal entropy, not towards it, right?

Emile, you've mixed up "optimization process" and "intelligence". According to your post, Eliezer wouldn't consider evolution an optimization process. He does; he doesn't consider it intelligent.

Oh, and if something maximising entropy is a valid optimisation process, then surely everything is an optimisation process and the term becomes useless?

Not really. A salt crystal is not an optimisation process - at least not on conventional timescales. However, certainly optimisation is a widespread, "universal" phenomena - driving most change in the universe and all self-organising systems - including biological ones.

Optimisation processes lead (locally) away from maximal entropy, not towards it, right?

That description certainly fits a star. The star is locally ordered (it's a big, dense ball of matter), but it creates global disorder - in the form of heat and radiation.

However, I don't think there is a general statement you can make about what optimisation processes do to "local" entropy levels. For a counter-example, consider gas expanding to fill a box. That is surely an optimisation process, and we know what solution it will converge on. However, there is no associated "optimising agent" using a heat pump to perform work to execute the task - and consequently there are not really any "local" increases in entropy occurring.

there are not really any "local" increases in entropy occurring.

I meant "decreases", of course. There might be local decreases if self-organising systems formed (e.g. if the gas became turbulent, and formed eddies), but that gets us into the realm of pedantic nit-picking.

Tim: I'm not offering a definition of "optimization process" or "intelligence", I'm just pointing out that when you say "gravity is an optimization process", or "stars are optimization processes", you're missing the whole point of the Kasparov post.

There's a difference between the unpredictability of a ball rolling down a hill, and the unpredictability of Kasparov's chess moves. This difference is not obvious.

Nick Tarleton: But then they're not optimizing for a goal they can be said to have. If a street is closed, they won't reach the destination, as they would if they knew what the destination was.

Neither will the person who has that goal, if they ignore the goal having made the plan.

People habitually overestimate the necessity of planning. Planning can be useful, but it is never enough, often unnecessary, and sometimes actively harmful.

You know, I've been thinking that I'm having trouble saying exactly what a "preference" is.

"that which an optimization process selects for" would seem to more or less declare everything an optimization process.

Anyone got a decent reduction on the notion of a preference itself? I'm kinda annoyed that I don't seem to be able to quite work that out.

I'm just pointing out that when you say [...] "stars are optimization processes", you're missing the whole point of the Kasparov post.

I was addressing the sentence: "But though stars burn longer and brighter than any craft of biology or human artifice, stars are neither optimization processes, nor products of strong optimization pressures." - which, IMHO, is factually inaccurate. I do not see any evidence that I am missing anything.

"Preferences" is just another name for the values of agents.

Tim Tyler:

Yes, I understood that. I meant can you reduce it to non-mind? Besides, evolution is an optimization process, and I wouldn't call it an agent that has values.

Either way, the point is that it's just moving my confusion from the word "preferences" to the word "values".

If I then say "those things an agent wants", then I'm just stuck at the word "want" again. It's not a reduction, just a repeated renaming of the same black box.

"Values" is another name for components of a utility function. One area where things get fuzzy is when you get to decide whether something qualifies as an agent or not - but that is a traditional problem which lies beyond the scope of this post.

Tim: no, I'd think of it in reverse, that a utility function is a very special type of encoding for a set of preferences.

Again, I'm not denying that I think I have an intuitive sense of what I think I mean by the term. It's just that when I try to reduce it from something mind to something non mind, the best I can come up with is stuff like "that which an optimization process selects for"

At which point I have to declare everything an optimization process in some sense. (=I'm actually semisorta tempted to do this, to talk about optimization power as a property of processes in general, rather than distinguishing certain types of processes as optimization processes. This way I think I'd have a reasonably serviceable reduction of the notion of a preference. Except then with intelligent agents that aren't logically omnicient and, say, can't yet fully compute their morality (or primality or whatever as appropriate) and thus in a sense don't actually fully know their preferences.

Well, there's hopefully enough here to illustrate my confusion sufficiently that you or someone who's actually worked out the correct answer can help me out here. I'm annoyed that I don't know this. :)

FYI -- the best chess player of the past 10 years has been a guy called Anand, and he is the best rapid chess player of all time. Maybe some day people will stop talking about Kasparov who retired a few years ago!

I was brought to full view of the picture in your last paragraph and I have to say.....You could have made this much shorter and much easier to understand (Especially for those "lacking"). Otherwise about half of that article is just you spinning your wheels going nowhere. Don't get me wrong, the idea is interesting, but for the most part it could have been expressed with,

"There are an infinite amount of possibilities for anything to happen. That said, in our day to day routines only our own intelligence,experiences, and knowledge can determine the amount of success that we have in our lives."

Right? It's similar to poetry, as I say, "If it has to be explained to you by someone more knowledgeable then it defeats the point of expressing yourself"

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30