« Competent Elites | Main | Friedman's "Prediction vs. Explanation" »

September 28, 2008

Comments

This post highlights an important disagreement I have with Eliezer.

Eliezer thinks that a group of AI scientists may be dangerous, because they aren't smart enough to make a safe AI.

I think that Eliezer is dangerous, because he thinks he's smart enough to make a safe AI.

"I think that Eliezer is dangerous, because he thinks he's smart enough to make a safe AI."

As far as I can tell, he's not going to go and actually make that AI until he has a formal proof that the AI will be safe. Now, because of the verification problem, that's no surefire guarantee that it will be safe, but it makes me pretty comfortable.

Vassar wrote:


***
I think it somewhat unlikely there are creationists at your level (Richard Smalley included) and would be astounded if there were any at mine. Well... I mean avowed and sincere biblical literalists, there might be all sorts of doctrines that could be called creationist.
***

I have no clear idea what you mean by "level" in the above...

IQ?

Demonstrated scientific or mathematical accomplishments?

Degree of agreement with your belief system? ;-)

-- Ben G

When Scott Aaronson was 12 years old, he: "set myself the modest goal of writing a BASIC program that would pass the Turing Test by learning from experience and following Asimov's Three Laws of Robotics..."

As I think back on that episode, I realize that even at the time, I didn't really expect to succeed; I just wanted to see how far I could get and what would happen if I tried. And it's not clear to me in retrospect that it wasn't worth a day's work: at the least, I learned something about how to write tokenizers and user interfaces! Certainly I've spent many, many days less usefully. For similar reasons, it's probably worth it for budding computer scientists to spend a few days on the P vs. NP question, even if their success probability is essentially zero: it's the only way to get a gut, intuitive feel for why the problem is hard.

Is it likewise possible that some of the AGI researchers you've met (not the creationist, but the other ones) aren't quite as stupid as they seemed? That even if they don't succeed at their stated goal (as I assume they won't), the fact that they're actually building systems and playing around with them makes it halfway plausible that they'll succeed at something?

Scott, if the question you're asking is "Can they learn something by doing this?" and not "Can they build AGI?" or "Can they build FAI?" a whole different standard applies. You can also learn something by trying to take apart an alarm clock.

Much of life consists of holding yourself to a high enough standard that you actually make an effort. If you're going to learn, just learn - get a textbook, try problems at the appropriate difficult-but-not-too-hard level. If you're going to set out to accomplish something, don't bait-and-switch to the "Oh, but I'll learn something even if I fail" when it looks like you might fail. Yoda was right: If you're going to do something, set out to do it, don't set out to try.

Eliezer: I'm pretty sure that MANY very smart people learn more from working on hard problems and failing quite frequently than from reading textbooks and practicing easy problems. Both should be part of an intellectual diet.

"I think that Eliezer is dangerous, because he thinks he's smart enough to make a safe AI."

As far as I can tell, he's not going to go and actually make that AI until he has a formal proof that the AI will be safe. Now, because of the verification problem, that's no surefire guarantee that it will be safe, but it makes me pretty comfortable.


Good grief.

Considering the nature of the problem, and the nature of Eliezer, it seems more likely to me that he will convince himself that he has proven that his AI will be safe, than that he will prove that his AI will be safe. Furthermore, he has already demonstrated (in my opinion) that he has higher confidence than he should that his notion of "safe" (eg., CEV) is a good one.

Many years ago, I made a mental list of who, among the futurists I knew, I could imagine "trusting" with godlike power. At the top of the list were Anders Sandberg and Sasha Chislenko. This was not just because of their raw brainpower - although they are/were in my aforementioned top ten list - but because they have/had a kind of modesty, or perhaps I should say a sense of humor about life, that would probably prevent them from taking giant risks with the lives of, and making decisions for, the rest of humanity, based on their equations.

Eliezer strikes me more as the kind of person who would take risks and make decisions for the rest of humanity based on his equations.

To phrase this in Bayesian terms, what is the expected utility of Eliezer creating AI over many universes? Even supposing he has a higher probability of creating beneficial friendly AI than anyone else, that doesn't mean he has a higher expected utility. My estimation is that he excels on the upside - which is what humans focus on - having a good chance of making good decisions. But my estimation is also that, in the possible worlds in which he comes to a wrong conclusion, he has higher chances than most other "candidates" do of being confident and forging ahead anyway, and of not listening to others who point out his errors. It doesn't take (proportionally) many such possible worlds to cancel out the gains on the upside.

So given EY's post about supernaturalism, if reductionism is true then we can't imagine anything that truly forms non reducible strata, there is only one strata that composes everything. Presumably, the laws this strata follow are supposed to be mathematical.

Consequently, Godel's incompleteness theorem (GIT) soundly refutes reductionism, right? I know GIT is becoming cliche and all, so bringing it up as a counter is perceived as anti-intellectual, but really, you can't just say a problem ceases to be a problem if it gets repeated too much.

Phil, your analysis depends a lot on what the probabilities are without Eliezer.

If Eliezer vanished, what probabilities would you assign to: (A) someone creating a singularity that removes most/all value from this part of the universe; (B) someone creating a positive singularity; (C) something else (e.g., humanity staying around indefinitely without a technological singularity)? Why?

There is a terrible complacency among people who have assimilated the ontological perspectives of mathematical physics and computer science, and the people who do object to the adequacy of naturalism are generally pressing in a retrograde direction.
Elaborate, please?

Anna, I haven't assigned probabilities to those events. I am merely comparing Eliezer to various other people I know who are interested in AGI. Eliezer seems to think that the most important measure of his ability, given his purpose, is his intelligence. He scores highly on that. I think the appropriate measure is something more like [intelligence * precision / (self-estimate of precision)], and I think he scores low on that relative to other people on my list.

Phil, that penalizes people who believe themselves to be precise even when they're right. Wouldn't, oh, intelligence / (1 + |precision - (self-estimate of precision)|) be better?

What do you mean by "precision", anyway?

Re: GIT - the main connections I see between Godel's incompleteness theorem and AI are that Hofstadter was interested in both, and Penrose was confused about both. What does it have to do with reductionism?

Phil, that penalizes people who believe themselves to be precise even when they're right. Wouldn't, oh, intelligence / (1 + |precision - (self-estimate of precision)|) be better?
Look at my little equation again. It has precision in the numerator, for exactly that reason.
What do you mean by "precision", anyway?

Precision in a machine-learning experiment (as in "precision and recall") means the fraction of the time that the answer your algorithm comes up with is a good answer. It ignores the fraction of the time that there is a good answer that your algorithm fails to come up with.

Phil: Your estimate rewards precision and penalizes self estimate of precision. A person of a given level of precision should be rewarded for believing their precision to be what it is, not for believing it to be low. If you had self-estimate of precision in the numerator that would negate Nick's claim, but then you could drop the term from both sides.

Mike: You're right - that is a problem. I think that in this case, underestimating your own precision by e is better than overestimating your precision by e (hence not using Nick's equation).

But it's just meant to illustrate that I consider overconfidence to be a serious character flaw in a potential god.

Phil, you might already understand, but I was talking about formal proofs, so your main worry wouldn't be the AI failing, but the AI succeeding at the wrong thing. (I.e., your model's bad.) Is that what your concern is?

Besides A2I2, what companies are claiming they'll reach general intelligence in five years?

Phil, you might already understand, but I was talking about formal proofs, so your main worry wouldn't be the AI failing, but the AI succeeding at the wrong thing. (I.e., your model's bad.) Is that what your concern is?
Yes. Also, the mapping from the world of the proof into reality may obliterate the proof.

Additionally, the entire approach is reminiscent of someone in 1800 who wants to import slaves to America saying, "How can I make sure these slaves won't overthrow their masters? I know - I'll spend years researching how to make REALLY STRONG leg irons, and how to mentally condition them to lack initiative." That approach was not a good long-term solution.

Phil... I'm sorry, but that's an indescribably terrible analogy.

CFAI: Beyond the adversarial attitude

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31