« Wanting To Want | Main | Anti-Edgy »

October 28, 2008

Comments

Interesting idea... though I still think you're wrong to step away from anthropomorphism, and 'necessary and sufficient' is a phrase that should probably be corralled into the domain of formal logic.

And I'm not sure this adds anything to Sternberg and Salter's definition: 'goal-directed adaptive behavior'.

I'm not an AGI researcher or developer (yet), but I think that the notion of a process steering the future into a constrained region is brilliant. It immediately feels much closer to implementation than any other definitions I've read before. Please continue posting on this topic. What I'm especially looking forward is anything on compression / abstraction over the search space.

Eliezer

One of your best ever posts IMHO and right on the nail. Of course this might be because I already agree with this definition but I left AI years ago, long before blogging and never wrote something like this up nor would I have said it so eloquently if I had.

Thom Blake
'goal-directed adaptive behaviour' is quite different and specifically it does not capture the notion of efficiency - of doing more with less.

Yep, definitely one of Eliezer's best posts.

Eliezer, out of curiosity, what was Lanier's response? Did he bite the bullet and say that wouldn't be an intelligence?

If you want to measure the intelligence of a system, I would suggest measuring its optimization power as before, but then dividing by the resources used. Or you might measure the degree of prior cognitive optimization required to achieve the same result using equal or fewer resources.

This really won't do. I think what you want is think in terms of a production function, which describe a system's output on a particular task as a function of its various inputs and features. Then we can talk about partial derivatives; rates at which output increases as a function of changes in inputs or features. The hard thing here is how to abstract well, how to collapse diverse tasks into similar task aggregates, and how to collapse diverse inputs and features into related input and feature aggregates. In particular, there is the challenge of how to identify which of those inputs or features count as its "intelligence."

IQ is a feature of human brains, and many have studied how the output of humans vary with IQ and task, both when other inputs are in some standard "reasonable" range, and when other inputs vary substantially. Even this is pretty hard; it is not at all clear to me how to broaden these abstractions even further, to talk about the "intelligence" of arbitrary systems on very wide ranges of tasks with wide ranges of other inputs and features.

I love this post, but there are several things in it that I have to take issue with.

You give the example of the asteroid and the tiger as things with different levels of intelligence, making the point that in both cases human beings are able to very rapidly respond with things that are outside the domain under which these entities are able to optimize. In the case of the asteroid, it can't optimize at all. In the case of the tiger, the human is more intelligent and so is better able to optimize over a wider domain.

However, the name of this blog is "overcoming bias," and one of its themes is the areas under which human intelligence breaks down. Just like you can toss poisoned meat to a tiger, there are things you can do to humans with similar effect. You can, for example, play on our strong confirmation bias. You can play tricks on our mind that exploit our poor intuition for statistics. Or, as in the case of Kasparov vs. Deep Blue, you can push us into a domain where we are slower than our opponent. Poisoned meat might do it for a tiger, but are economic bubbles and the "greater fool" pyramid that builds within them any different than this for humans? Aren't opaque mortgage-backed securities like the poisoned meat you toss to the tiger?

Secondly, I often object to the tendency of CS/AI folks to claim that evolution is incredibly slow. Simple iterative mutation and selection is slow and blind, but most of what goes on in evolution is not simple iterative mutation and selection. Evolution has evolved many strategies for evolution-- this is called the evolution of evolvability in the literature. These represent strategies for more efficiently finding local maxima in the fitness landscape under which these evolutionary processes operate. Examples include transposons, sexual reproduction, stress-mediated regulation of mutation rates, the interaction of epigenetics with the Baldwin effect, and many other evolvability strategies. Evolution doesn't just learn... it learns how to learn as well. Is this intelligence? It's obviously not human-like intelligence, but I think it qualifies as a form of very alien intelligence.

Finally, the paragraph about humans coming to parity with nature after 500 years of modern science is silly. Parity implies that there is some sort of conflict or contest going on. We are a part of the natural system. When we build roads and power plants, make vaccines, convert forests to farmland, etc. we are not fighting nature. We are part of nature, so these things are simply nature modifying itself just as it always has. Nuclear power plants are as "natural" as beehives and beaver dams. The "natural" vs. "artificial" dichotomy is actually a hidden form of anthropomorphism. It assumes that we are somehow metaphysically special.

g is a feature of human brains; IQ is a rough ranking of human brains with respect to g. I have not yet read of any means of actually measuring g , has anyone here got any references?

Eliezer, I have been trying to reread your series of ethics and morality posts in order, but am having trouble following the links backwards, I keep "finding" posts I missed. Any chance you could go and link them in the order you think they should be read?

Billswift, re rereading the series, check out Andrew Hay's list and associated graphs.

http://www.google.com/search?hl=en&q=tigers+climb+trees

On a more serious note, you may be interested in Marcus Hutter's 2007 paper "The Loss Rank Principle for Model Selection". It's about modeling, not about action selection, but there's a loss function involved, so there's a pragmatist viewpoint here, too.

Adam_Ierymenko: Evolution has evolved many strategies for evolution-- this is called the evolution of evolvability in the literature. These represent strategies for more efficiently finding local maxima in the fitness landscape under which these evolutionary processes operate. Examples include transposons, sexual reproduction,

Yes, Eliezer_Yudkowsky has discussed this before and calls that optimizaiton at the meta-level. Here is a representative post where he makes those distinctions.

Looking over the history of optimization on Earth up until now, the first step is to conceptually separate the meta level from the object level - separate the structure of optimization from that which is optimized.

If you consider biology in the absence of hominids, then on the object level we have things like dinosaurs and butterflies and cats. On the meta level we have things like natural selection of asexual populations, and sexual recombination.

A quote from that post: "So animal brains - up until recently - were not major players in the planetary game of optimization; they were pieces but not players."

Again, no mention of sexual selection. Brains are players in the optimisation process. Animal brains get to perform selection directly. Females give male tyres a good kicking before choosing them - a bit like unit testing. Sexual selection is not even the only mechanism - natural selection can do this too.

Adam, you will find above that I contrasted human design to biology, not to nature.

Trying to toss a human a poisoned credit-default swap is more like trying to outrun a tiger or punch it in the nose - it's not an Outside Context Problem where the human simply doesn't understand what you're doing; rather, you're opposing the human on the same level and it can fight back using its own abilities, if it thinks of doing so.

Robin: This really won't do. I think what you want is think in terms of a production function, which describe a system's output on a particular task as a function of its various inputs and features. Then we can talk about partial derivatives; rates at which output increases as a function of changes in inputs or features. The hard thing here is how to abstract well, how to collapse diverse tasks into similar task aggregates, and how to collapse diverse inputs and features into related input and feature aggregates. In particular, there is the challenge of how to identify which of those inputs or features count as its "intelligence."

How do you measure output? As a raw quantity of material? As a narrow region of outcomes of equal or higher preference in an outcome space? Economists generally deal in quantities that are relatively fungible and liquid, but what about when the "output" is a hypothesis, a design for a new pharmaceutical, or an economic rescue plan? You can say "it's worth what people will pay for it" but this just palms off the valuation problem on hedge-funds or other financial actors, which need their own way of measuring the value somehow.

There's also a corresponding problem for complex inputs. As economists, you can to a large extent sit back and let the financial actors figure out how to value things, and you just measure the dollars. But the AI one tries to design is more in the position of actually being a hedge fund - the AI itself has to value resources and value outputs.

Economists tend to measure intermediate tasks that are taken for granted, but one of the key abilities of intelligence is to Jump Out Of The System and trace a different causal pathway to terminal values, eliminating intermediate tasks along the way. How do you measure fulfillment of terminal values if, for example, an AI or economy decides to eliminate money and replace it with something else? We haven't always had money. And if there's no assumption of money, how do you value inputs and outputs?

You run into problems with measuring the improbability of an outcome too, of course; I'm just saying that breaking up the system into subunits with an input-output diagram (which is what I think you're proposing?) is also subject to questions, especially since one of the key activities of creative intelligence is breaking obsolete production diagrams.

billswift, you might find this helpful:
http://www.cs.auckland.ac.nz/~andwhay/graphsfiles/dependencygraphs.html#Something%20to%20protect.dot

With all this talk about poisoned meat and CDSes, I was inspired to draw this comic.

It's interesting that Eliezer ties intelligence so closely to action ("steering the future"). I generally think of intelligence as being inside the mind, with behaviors & outcomes serving as excellent cues to an individual's intelligence (or unintelligence), but not as part of the definition of intelligence. Would Deep Blue no longer be intelligent at chess if it didn't have a human there to move the pieces on the board, or if it didn't signal the next move in a way that was readily intelligible to humans? Is the AI-in-a-box not intelligent until it escapes the box?

Does an intelligent system have to have its own preferences? Or is it enough if it can find the means to the goals (with high optimization power, across domains), wherever the goals come from? Suppose that a machine was set up so that a "user" could spend a bit of time with it, and the machine would figure out enough about the user's goals, and about the rest of the world, to inform the user about a course of action that would be near-optimal according to the user's goals. I'd say it's an intelligent machine, but it's not steering the future toward any particular target in outcome space. You could call it intelligence as problem-solving.

Eliezer's comment describes the importance of Jumping Out Of The System, which I attribute to the "cross-domain" aspect of intelligence, but I don't see this defined anywhere in the *formula* given for intelligence, which so far only covers "efficient" and "optimizer".

First, a quick-and-dirty description of the process: Find an optimization process in domain A (whether or not it help attain goals). Determine one or many mapping functions between domains A and B. Use a mapping to apply the optimization process to achieve a goal in domain B.

I think the heart of crossing domains is in the middle step - the construction of a mapping between domains. Plenty of these mappings will be incomplete, mere projections that lose countless dimensions, but they still occasionally allow for useful portings of optimization processes. This is the same skill as abstraction or generalization: turning data into simplified patterns, turning apples and oranges into numbers all the same. The measure of this power could then be the maximum distance from domain A to domain B that the agent can draw mappings across. Or maybe the maximum possible complexity of a mapping function (or is that the same thing)? Or the number of possible mappings between A and B? Or speed; it just would not do to run through every possible combination of projections between two domains. So here, then, is itself a domain that can be optimized in. Is the measure of being cross-domain just a measure of how efficiently one can optimize in the domain of "mapping between domains"?

Would Deep Blue no longer be intelligent at chess if it didn't have a human there to move the pieces on the board, or if it didn't signal the next move in a way that was readily intelligible to humans?

With no actuators at all, how would you distinguish the intelligence of Deep Blue from that of a heavy inert metal box?

Eliezer, even if you measure output as you propose in terms of a state space reduction factor, my main point was that simply "dividing by the resources used" makes little sense. Yes a production function formulation may abstract from some relevant details, but it is far closer to reality than dividing by "resources." Yes a market economy may help one to group and measure relevant inputs, and without that aid you'll have even more trouble grouping and measuring inputs.

With no actuators at all, how would you distinguish the intelligence of Deep Blue from that of a heavy inert metal box?

One tree near my home is an excellent chess player. If only it had some way to communicate...

I could teach any deciduous tree to play grandmaster chess if only I could communicate with it. (Well, not the really dumb ones.)

I have not yet read of any means of actually measuring g , has anyone here got any references?

There's no way to "actually measure g", because g has no operational definition beyond statistical analyses of IQ.

There have been some attempts to link calculated g with neural transmission speeds and how easily brains can cope with given problems, but there's been little success.

Re resources: if you look at how IQ tests work they usually attempt to factor out resources as best they can - no mechanical aids allowed, fixed time, etc. The only "wealth" you are permitted
is what they can't strip off - education, genes, etc.

I imagine computer tests could be a bit like that - e.g. no net connection allowed, but you don't get penalised for having a faster CPU. For example, if you look at computer go contests, people get to supply their own hardware - and the idea is to win. You do not get many bonus points for using an Apple II.

Jeff Hawkins, in his book On Intelligence, says something similar to Eliezer. He says intelligence IS prediction. But Eliezer say intelligence is steering the future, not just predicting it. Steering is a behavior of agency, and if you cannot peer into the source code but only see the behaviors of an agent, then intelligence would necessarily be a measure of steering the future according to preference functions. This is behaviorism is it not? I thought behaviorism had been predicated as a useful field of inquiry in the cognitive sciences?

I can see where Eliezer is going with all this. The most moral/ethical/friendly AGI cannot take orders from any human, let alone be modeled on human agency to a large degree itself, and we also definitely do not want this agency to be a result of the same horrendous process of natural selection red in tooth and claw that created us.

That cancels out an anthropomorphic AI, cancels out evolution through natural selection, and it cancels out an unchecked oracle/genie type wish-granting intelligent system (though I personally feel that a controlled (friendly?) version of the oracle AI is the best option because I am skeptical with regard to Eliezer or anyone else coming up with a formal theory of friendliness imparted on an autonomous agent). ((Can an oracle type AI create a friendly AI agent? Is that a better path towards friendliness?))

Adam's comment above is misplace because I think Eliezer's recursively self-improving friendly intelligence optimization is a type of evolution, just not as blind as natural selection as has been played out through natural history on our earth.

Nice post.

And no, I don't think optimisation processes capture best what we generally mean by intelligence - but it captures best what we should mean by intelligence.

In practice it may have some weaknesses - depending on what exactly the valid domains are, there may be a more informative concept of intelligence in our universe - but it's good enough to do work with, while most definitions of intelligence aren't.

Would it do good to use something like sentience quotient, the quantity of bits per second per kg of matter a system can process, to assess the efficiency of a system ?

Of two systems having the same preferences, and the same sentience quotient, but whose optimization power isn't the same, one must then have a more efficient, smarter way of optimizing than the other ?

As for cross domain optimization, I don't see offhand how to mathematically charaterize different domains - and it is possible to define arbitrary domains anyway I think -

but if you have a nonrandom u, and are niverse, or environment, and are adapted to it, then if following your preferences you want to use all the information available locally in your environment, in your past light cone, you can only predict the course of your actions faster than will the universe given the implementation of physical laws upon matter if you can non destructively compress the information describing that environment; I guess, this works in any universe that has not reached maximal entropy, and the less entropy in that universe, the faster your speed for predicting future events will be compared to the speed of the universe implementing future events.

If you can't do that, then you have to use destructive compression to simplify your information about the environment into something you can manageably use to compute the future state of the universe following your actions, faster than the universe itself would implement them. There's a tradeoff between speed, simplicity, and precision, error rate in this case.

Just my immediate thoughts.

Jeff Hawkins observes that brains are constantly predicting the future.

That's quite consistent with the idea of brains acting as expected utility maximisers.

The brain predicts the future in order to detect if its model of the world needs updating when new sensory data arrives. If the data matches the model - no problem. If the data and the model conflict then the model needs updating.

In the expected utility framework, the brain has to predict the future anyway in order to judge the expected consequences of its actions. All it does is keep the results around for long enough to see if things go as it expected.

I often object to the tendency of CS/AI folks to claim that evolution is incredibly slow.

It is only nucleic evolution which is relatively slow. Cultural evolution has enormously magnified the rate of evolutionary change on the planet. In a century, skyscrapers have appeared out of nowhere, machines have probed the other planets, and the Earth has started to glow at night. Today, we can see evolutionary change taking place in real time - within an an individual's lifespan.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31