Is Morality Given?

Continuation of:  Is Morality Preference?

(Disclaimer:  Neither Subhan nor Obert represent my own position on morality; rather they represent different sides of the questions I hope to answer.)

Subhan:  "What is this 'morality' stuff, if it is not a preference within you?"

Obert:  "I know that my mere wants, don't change what is right; but I don't claim to have absolute knowledge of what is right -"

Subhan:  "You're not escaping that easily!  How does a universe in which murder is wrong, differ from a universe in which murder is right?  How can you detect the difference experimentally?  If the answer to that is 'No', then how does any human being come to know that murder is wrong?"

Obert:  "Am I allowed to say 'I don't know'?"

Subhan:  "No.  You believe now that murder is wrong.  You must believe you already have evidence and you should be able to present it now."

Obert:  "That's too strict!  It's like saying to a hunter-gatherer, 'Why is the sky blue?' and expecting an immediate answer."

Subhan:  "No, it's like saying to a hunter-gatherer:  Why do you believe the sky is blue?"

Obert:  "Because it seems blue, just as murder seems wrong.  Just don't ask me what the sky is, or how I can see it."

Subhan:  "But - aren't we discussing the nature of morality?"

Obert:  "That, I confess, is not one of my strong points.  I specialize in plain old morality.  And as a matter of morality, I know that I can't make murder right just by wanting to kill someone."

Subhan:  "But if you wanted to kill someone, you would say, 'I know murdering this guy is right, and I couldn't make it wrong just by not wanting to do it.'"

Obert:  "Then, if I said that, I would be wrong.  That's common moral sense, right?"

Subhan:  "Argh!  It's difficult to even argue with you, since you won't tell me exactly what you think morality is made of, or where you're getting all these amazing moral truths -"

Obert:  "Well, I do regret having to frustrate you.  But it's more important that I act morally, than that I come up with amazing new theories of the nature of morality.  I don't claim that my strong point is in explaining the fundamental nature of morality.  Rather, my strong point is coming up with theories of morality that give normal moral answers to questions like, 'If you feel like killing someone, does that make it right to do so?'  The common-sense answer is 'No' and I really see no reason to adopt a theory that makes the answer 'Yes'.  Adding up to moral normality - that is my theory's strong point."

Subhan:  "Okay... look.  You say that, if you believed it was right to murder someone, you would be wrong."

Obert:  "Yes, of course!  And just to cut off any quibbles, we'll specify that we're not talking about going back in time and shooting Stalin, but rather, stalking some innocent bystander through a dark alley and slitting their throat for no other reason but my own enjoyment.  That's wrong."

Subhan:  "And anyone who says murder is right, is mistaken."

Obert:  "Yes."

Subhan:  "Suppose there's an alien species somewhere in the vastness of the multiverse, who evolved from carnivores.  In fact, through most of their evolutionary history, they were cannibals.  They've evolved different emotions from us, and they have no concept that murder is wrong -"

Obert:  "Why doesn't their society fall apart in an orgy of mutual killing?"

Subhan:  "That doesn't matter for our purposes of theoretical metaethical investigation.  But since you ask, we'll suppose that the Space Cannibals have a strong sense of honor - they won't kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong.  Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies.  But so far as the actual killing is concerned, the aliens just think it's fun.  When someone gets executed for, say, driving through a traffic light, there's a bidding war for the rights to personally tear out the offender's throat."

Obert:  "Okay... where is this going?"

Subhan:  "I'm proposing that the Space Cannibals not only have no sense that murder is wrong - indeed, they have a positive sense that killing is an important part of life - but moreover, there's no path of arguments you could use to persuade a Space Cannibal of your view that murder is wrong.  There's no fact the aliens can learn, and no chain of reasoning they can discover, which will ever cause them to conclude that murder is a moral wrong.  Nor is there any way to persuade them that they should modify themselves to perceive things differently."

Obert:  "I'm not sure I believe that's possible -"

Subhan:  "Then you believe in universally compelling arguments processed by a ghost in the machine.  For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value -1.  A mind is a physical device and you can't have a little blue woman pop out of nowhere and make it say 1 when the physics calls for it to say 0."

Obert:  "Suppose I were to concede this.  Then?"

Subhan:  "Then it's possible to have an alien species that believes murder is not wrong, and moreover, will continue to believe this given knowledge of every possible fact and every possible argument.  Can you say these aliens are mistaken?"

Obert:  "Maybe it's the right thing to do in their very different, alien world -"

Subhan:  "And then they land on Earth and start slitting human throats, laughing all the while, because they don't believe it's wrong.  Are they mistaken?"

Obert:  "Yes."

Subhan:  "Where exactly is the mistake?  In which step of reasoning?"

Obert:  "I don't know exactly.  My guess is that they've got a bad axiom."

Subhan:  "Dammit!  Okay, look.  Is it possible that - by analogy with the Space Cannibals - there are true moral facts of which the human species is not only presently unaware, but incapable of perceiving in principle?  Could we have been born defective - incapable even of being compelled by the arguments that would lead us to the light?  Moreover, born without any desire to modify ourselves to be capable of understanding such arguments?  Could we be irrevocably mistaken about morality - just like you say the Space Cannibals are?"

Obert:  "I... guess so..."

Subhan:  "You guess so?  Surely this is an inevitable consequence of believing that morality is a given, independent of anyone's preferences!  Now, is it possible that we, not the Space Cannibals, are the ones who are irrevocably mistaken in believing that murder is wrong?"

Obert:  "That doesn't seem likely."

Subhan:  "I'm not asking you if it's likely, I'm asking you if it's logically possible!  If it's not possible, then you have just confessed that human morality is ultimately determined by our human constitutions.  And if it is possible, then what distinguishes this scenario of 'humanity is irrevocably mistaken about morality', from finding a stone tablet on which is written the phrase 'Thou Shalt Murder' without any known justification attached?  How is a given morality any different from an unjustified stone tablet?"

Obert:  "Slow down.  Why does this argument show that morality is determined by our own constitutions?"

Subhan:  "Once upon a time, theologians tried to say that God was the foundation of morality.  And even since the time of the ancient Greeks, philosophers were sophisticated enough to go on and ask the next question - 'Why follow God's commands?'  Does God have knowledge of morality, so that we should follow Its orders as good advice?  But then what is this morality, outside God, of which God has knowledge?  Do God's commands determine morality?  But then why, morally, should one follow God's orders?"

Obert:  "Yes, this demolishes attempts to answer questions about the nature of morality just by saying 'God!', unless you answer the obvious further questions.  But so what?"

Subhan:  "And furthermore, let us castigate those who made the argument originally, for the sin of trying to cast off responsibility - trying to wave a scripture and say, 'I'm just following God's orders!'  Even if God had told them to do a thing, it would still have been their own decision to follow God's orders."

Obert:  "I agree - as a matter of morality, there is no evading of moral responsibility.  Even if your parents, or your government, or some kind of hypothetical superintelligence, tells you to do something, you are responsible for your decision in doing it."

Subhan:  "But you see, this also demolishes the idea of any morality that is outside, beyond, or above human preference.  Just substitute 'morality' for 'God' in the argument!"

Obert:  "What?"

Subhan:  "John McCarthy said:  'You say you couldn't live if you thought the world had no purpose. You're saying that you can't form purposes of your own—that you need someone to tell you what to do. The average child has more gumption than that.'  For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question:  'And why obey this morality?'  It would be your decision to follow this trend of the universe, or obey this structure of logic.  Your decision - and your preference."

Obert:  "That doesn't follow!  Just because it is my decision to be moral - and even because there are drives in me that lead me to make that decision - it doesn't follow that the morality I follow consists merely of my preferences.  If someone gives me a pill that makes me prefer to not be moral, to commit murder, then this just alters my preference - but not the morality; murder is still wrong.  That's common moral sense -"

Subhan:  "I beat my head against my keyboard!  What about scientific common sense?  If morality is this mysterious given thing, from beyond space and time - and I don't even see why we should follow it, in that case - but in any case, if morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

Obert:  "Coincidence?  How so?"

Subhan:  "Just where on Earth do you think the emotion of love comes from?  If the ancient Greeks had ever thought of the theory of natural selection, they could have looked at the human institution of sexual romance, or parental love for that matter, and deduced in one flash that human beings had evolved - or at least derived tremendous Bayesian evidence for human evolution.  Parental bonds and sexual romance clearly display the signature of evolutionary psychology - they're archetypal cases, in fact, so obvious we usually don't even see it."

Obert:  "But love isn't just about reproduction -"

Subhan:  "Of course not; individual organisms are adaptation-executers, not fitness-maximizers.  But for something independent of humans, morality looks remarkably like godshatter of natural selection.  Indeed, it is far too much coincidence for me to credit.  Is happiness morally preferable to pain?  What a coincidence!  And if you claim that there is any emotion, any instinctive preference, any complex brain circuitry in humanity which was created by some external morality thingy and not natural selection, then you are infringing upon science and you will surely be torn to shreds - science has never needed to postulate anything but evolution to explain any feature of human psychology -"

Obert:  "I'm not saying that humans got here by anything except evolution."

Subhan:  "Then why does morality look so amazingly like a product of an evolved psychology?"

Obert:  "I don't claim perfect access to moral truth; maybe, being human, I've made certain mistakes about morality -"

Subhan:  "Say that - forsake love and life and happiness, and follow some useless damn trend of the universe or whatever - and you will lose every scrap of the moral normality that you once touted as your strong point.  And I will be right here, asking, 'Why even bother?'  It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow."

Obert:  "All right... then maybe the reason morality seems to bear certain similarities to our human constitutions, is that we could only perceive morality at all, if we happened, by luck, to evolve in consonance with it."

Subhan:  "Horsemanure."

Obert:  "Fine... you're right, that wasn't very plausible.  Look, I admit you've driven me into quite a corner here.  But even if there were nothing more to morality than preference, I would still prefer to act as morality were real.  I mean, if it's all just preference, that way is as good as anything else -"

Subhan:  "Now you're just trying to avoid facing reality!  Like someone who says, 'If there is no Heaven or Hell, then I may as well still act as if God's going to punish me for sinning.'"

Obert:  "That may be a good metaphor, in fact.  Consider two theists, in the process of becoming atheists.  One says, 'There is no Heaven or Hell, so I may as well cheat and steal, if I can get away without being caught, since there's no God to watch me.'  And the other says, 'Even though there's no God, I intend to pretend that God is watching me, so that I can go on being a moral person.'  Now they are both mistaken, but the first is straying much further from the path."

Subhan:  "And what is the second one's flaw?  Failure to accept personal responsibility!"

Obert:  "Well, and I admit I find that a more compelling argument than anything else you have said.  Probably because it is a moral argument, and it has always been morality, not metaethics, with which I claimed to be concerned.  But even so, after our whole conversation, I still maintain that wanting to murder someone does not make murder right.  Everything that you have said about preference is interesting, but it is ultimately about preference - about minds and what they are designed to desire - and not about this other thing that humans sometimes talk about, 'morality'.  I can just ask Moore's Open Question:  Why should I care about human preferences?  What makes following human preferences right?  By changing a mind, you can change what it prefers; you can even change what it believes to be right; but you cannot change what is right.  Anything you talk about, that can be changed in this way, is not 'right-ness'."

Subhan:  "So you take refuge in arguing from definitions?"

Obert:  "You know, when I reflect on this whole argument, it seems to me that your position has the definite advantage when it comes to arguments about ontology and reality and all that stuff -"

Subhan:  "'All that stuff'?  What else is there, besides reality?"

Obert:  "Okay, the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks.  But I still think the morality-as-given viewpoint has the advantage when it comes to, you know, the actual morality part of it - giving answers that are good in the sense of being morally good, not in the sense of being a good reductionist.  Because, you know, there are such things as moral errors, there is moral progress, and you really shouldn't go around thinking that murder would be right if you wanted it to be right."

Subhan:  "That sounds to me like the logical fallacy of appealing to consequences."

Obert:  "Oh?  Well, it sounds to me like an incomplete reduction - one that doesn't quite add up to normality."

Comments

[Dynamically Linked]

Subhan: "You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No', then how does any human being come to know that murder is wrong?"
...
Obert: "Because it seems blue, just as murder seems wrong. Just don't ask me what the sky is, or how I can see it."

But we already know why murder seems wrong to us. It's completely explained by a combination of game theory, evolutionary psychology, and memetics. These explanations screen off our apparent moral perceptions from any other influence. In order words, conditioned on these explanations being true, our moral perceptions are independent of (i.e. uncorrelated with) any possible morality-as-given, even if it were to exist.

So there is a stronger argument against Obert than the one Subhan makes. It's not just that we don't know how we can know about what is right, but rather that we know we can't know, at least not through these apparent moral perceptions/intuitions.

[robin brandt]

As with human aesthetic sense, human morality may be approximations and versions of more absolutely definable optimal solutions to information-theoretic, game-theoretic, social, economic, intelligence, signaling, cooperation problems. Therefore it may be likely that an alien race could share some of the same values as we do, because it may turn out that they are "good" solutions for intelligent culture bearing species in general. But there is nothing in the universe in it self that says that theese optimal solutions, or any value what-so-ever can be valuable, and I don´t understand why some atheists even expect there to be something like this. It is minds that give value to certain phenomena, and it usually happens because our emotional circuitry where wired to value something by evolution.

But I so no problem in trying to choose some of the more general looking values evolution has given us and making them even more general and refined. We should do this in such a way as to keep us stable and happy but also so that we have a rich future mind-space to move in, and I think we have already done this in part with Goodness, Truth and Beauty for it´s own sake, but they are not easily definable. It is not easy to find a sweet-spot between a species-specific version and a more general one that is free and pleasing to as many minds as possible. I think we should continue to refine this sweet-spot and work towards some values for their own sake and not only because they give us pleasure. I think this is important and can add up to the stability of an individual as well as a society and an AI. It certainly is a dangerous thing, but I see it as essential at least for a human mind.

One problem may be though that we are wired so that, if we don´t believe that morality or some other value is intrinsically rooted and valued by some great preferably immortal and omnipresent authority(God, the Universe, Nature) then we may have some trouble behaving morally or in accordance with that value, just because we choose to do so. Some strong people may find this easy. But many finds this very hard, and I think that is the prime reason why religion still persists today, even though most people know somewhere deep down that it is a dead practice.
And I will admit although not with any pride that I, myself has great trouble with pursuing and actually doing what I value, systematically and with pleasure and discipline, even though I know about all this. It may be that something is particularly wrong in my brain, or it may be quite a wide problem.

Thank you Eli for keeping on enlightening my path, day after day, I have been lurking on you for 2.5 years now, you have totally changed my life, before you, there was no person I could really trust in deep matters, and now, because you are so often right and I have to be so careful with trusting you, I have also become extremely careful in trusting my own intuitions and thought patterns. You learn people to think with rigorous self-critique, extream precision and dedication and mindful focus on the actual territory and possibility-space of study without getting trapped in the usual mind projection fallacy and other biases.

Keep on fighting! Your books will sell well, and it will fuel your goals!

[Dynamically Linked]

And to answer Obert's objection that Subhan's position doesn't quite add up to normality: before we knew game theory, evolutionary psychology, and memetics, nothing screened off our moral perceptions/intuitions from a hypothesized objective moral reality, so that was perhaps the best explanation available, given what we knew back then. And since that was most of human history, it's no surprise that morality-as-given feels like normality. But given what we know today, does it still make sense to insist that our meta-theory of morality add up to that normality?

[robin brandt]

I will try to express some of my points more accurately...
A human value, may it concern, knowledge, morality or beauty gets it's meaning from it's emotional base although it may be a frequent value in the space of possible intelligent species. Only minds can attribute value to something. The thing it attributes it to, may be universal or specific, but the thing itself is can not be valued by something other than a mind. To value something is a cognitive, emotional process, not some intrinsic property of some phenomenon.
But to believe this as the mind you are may not be the best way to achieve what you value and desire. We seem to work most efficiently towards a value when we believe it is intrinsically true and the only way. It may slow down that process considerably to know that values can´t be rooted in something outside other minds. It may also be liberating knowledge, and may fuel your productivity and mood. It may depend on your starting assumptions and expectations concerning values in general. My solution is to pick some very general values after serious consideration, and then to start to almost religiously work towards optimizing them, being open and critical of everything else, without ever introducing other magical thinking or supernatural phenomena, just trying to hack my own mind.

[michael vassar]

I think that we need a much better explanation of this word "mind". Supposedly mind space contains a -1 for every 1, but that simply sounds like system space.
I honestly think that the ontology has to go deeper here before progress is possible. Similar problem to born postulates and why we aren't Boltzman Brains.

[Caledonian]

Similar problem to born postulates and why we aren't Boltzman Brains.

We are Boltzmann Brains. You simply don't appreciate what restrictions are inherent in specifying the subset of Brains that can be called "we".

Not that this has anything to do with the topic, which everyone is very carefully skating around without addressing: what are operational definitions for right and wrong? When Obert says "Because it seems blue, just as murder seems wrong.", what collection of properties does wrong refer to? For that matter, what does blue refer to?

These questions have very simple and obvious answers which you will never grasp until you force yourself to face the questions. You mean something when you use the terms - you already recognize what is implied when you or someone else uses those terms. Now make that recognition explicit instead of implicit.

Do not "philosophize". That is attempting to understand the territory by making a diagram of map-making. It's adding another layer of analysis between you and the core concept, like an oyster adding a layer of nacre around an irritating bit of sand. You do not need a more complex ontology - you need to abolish the ontology.

[Richard]

I don't think you have to postulate Space Cannibals in order to imagine rational creatures who don't think murder is wrong. For a recent example, consider Rwanda 1994.

And I think it's quite possible that there might exist moral facts which humans are incapable of perceiving. We aren't just universal Turing machines, after all. Billions of years of evolution might produce creatures with moral blind spots, anologous to the blind spot in the human eye. Just as the squid's eye has no blind spot, a different evolutionary path might produce creatures with a greater or lesser innate capacity to perceive goodness than ourselves.

[Caledonian]

Maybe this will make it easier:

Obert says "just as murder seems wrong". There is a redundancy in that phrase. What is the redundancy, and why doesn't Obert perceive it as one?

What is the difference between saying something is a rube and not a blegg, and saying that someone appears to be a rube and not a blegg?

What is the difference between saying something is imperceivable, and saying something appears to be imperceivable?

[Z. M. Davis]

The notion of morality as subjectively objective computation seems a lot closer to Subhan's position than Obert's.

[Phillip Huggan]

Yes, EY's past positions about Morality are closer to Subhan's than Obert's. But AGI is software programming and hardware engineering, not being a judge or whoever writes laws.
I wouldn't suggest deifying EY if your goal is to learn ethics.

[Constant]

But we already know why murder seems wrong to us. It's completely explained by a combination of game theory, evolutionary psychology, and memetics. These explanations screen off our apparent moral perceptions from any other influence. In order words, conditioned on these explanations being true, our moral perceptions are independent of (i.e. uncorrelated with) any possible morality-as-given, even if it were to exist.

Let's try the argument with mathematics: we know why we think 5 is a prime number. It's completely explained by our evolution, experiences, and so on. Conditioned on these explanations being true, our mathematical perceptions are independent of mathematical-truth-as-given, even if it were to exist.

The problem is that mathematical-truth-as-given may shape the world and therefore shape our experiences. That is, we may have had the tremendous difficulty we had in factorizing the number 5 precisely because the number 5 is in fact a prime number. So one place where one could critique your argument is in the bit that goes: "conditioned on X being the case, then our beliefs are independent of Y". The critique is that X may in fact be a consequence of Y, in which case X is itself not independent of Y.

[Z. M. Davis]

"But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure.

[Robin Hanson]

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

[Ben Jones]

"You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No'...

Minor quibble - 'no' is not a sensical answer to any of those questions. Possibly remove the word 'how' from one of them?

Once again, no revelations that I haven't come across on my own, but crystallised and clarified brilliantly. Looking forward to the next few.

[marcio rpsbrk]

It seems to me that Obert makes a faulty interpretation of "there is no reason to talk about a 'morality' distinct from what people want.", but i would like to know what the author thinks. In my view, that assertion says not that ALL MORAL CLAIMS ARE WHIMS, but instead that to understand and parse and compare moral claims we have to resort to wants. In other words, that WANTS ARE THE OBJECT OF MORALITY, THOUGH NOT IT'S MATTER. To understand any moral claim we have to consider how it imparts onto what real, concrete persons feel and desire.

"I want pie" and "I deserve pie" are different, but i don't see how Subhan's arguments aspire to make them equal.

[Psy-Kosh]

Obert's arguments seem much closer to "how it feels from the inside", Subhan in general does seem to have stronger actual arguments, however:

"For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question: 'And why obey this morality?'" This, to me, smells of zombieism. "for any configuration of matter/energy/whatever, we can ask 'and why should we believe that this is actually conscious rather than just a structure immitating a consciousness?'"

[Phillip Huggan]

(ZMDavis wrote:) "But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure."

Argh. I didn't mean that as a critique on EY's prowess as an AGI theorist or programmer. I doubt Jesus would've wanted people to deify him, just to be nice to eachother. I doubt EY meant for his learning of philosophy to be interpreted as some sort of Moral code, he was just arrogant enough not to state he was sometimes using his list to as a tool to develop his own philosophy. I'm assuming any AGI project would be a team, and I'd doubt he'd challenge his best comparitive advantage is not ethics. Maybe he plans on writing the part of the code that tells an AGI how to stop using resources for a given job.

[Paul Gowder]

So here's a question Eliezer: is Subhan's argument for moral skepticism just a concealed argument for universal skepticism? After all, there are possible minds that do math differently, that do logic differently, that evaluate evidence differently, that observe sense-data differently...

Either Subhan can distinguish his argument from an argument for universal skepticism, or I say that it's refuted by reductio, since universal skepticism fails to the complete impossibility of asserting it consistently + things like moorean facts.

[Z. M. Davis]

Phillip, you're the one who brought up "deification," in response to my one-line comment, which you seem to have read a lot into. My second comment was intended to be humorous. I apologize for the extent to which I contributed to this misunderstanding.

[Don Geddis]

Eliezer seems to suggest that the only possible choices are morality-as-preference or morality-as-given, e.g. with reasoning like this:

[...] the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks. But I still think the morality-as-given viewpoint has the advantage [...]

But really, evolutionary psychology, plus some kind of social contract for group mutual gain, seems to account for the vast bulk of what people consider to be "moral" actions, as well as the conflict between private individual desires vs. actions that are "right". (People who break moral taboos are viewed not much differently from traitors in wartime, who betray their team/side/cause.)

I don't understand this series. Eliezer is writing multiple posts about the problems with the metatheories of morality as either preferences or given. Sure, both those metatheories are wrong. Is that really so interesting? Why not start to tackle what morality actually is, rather than merely what it is not?

[George Weinberg]

I think it's probably useful to taboo the word "should" for this discussion. I think when people say you "should" do X rather than Y it means something like "experience indicates X is more likely to lead to a good outcome than Y". People tend to have rule-based rather than consequence based moral systems because the full consequences of one's actions are unforeseeable. A rule like "one shouldn't lie" comes about because experience has shown that lying often has negative consequences for the speaker and listener and possibly others as well, although the particular consequences of a particular lie may be unforeseeable.

I don't see how there can be agreement as to moral principles unless there is first a reasonably good agreement as to what constitutes good and bad final states.

[Ian C.]

Relationships are real. For example if a plant is "under" a table, that is a fact, not a subjective whim of the observer. So if morality is a relationship, then aliens and man can have different moralities but both be objective, not subjective. The relationship would be between the object sought and the entity seeking it, e.g. murder + man = bad, murder + alien = good.

[Aleksei Riikonen]

Paul Gowder,

Yes, there are possible minds that do math/logic/deduction differently. Most of these logically possible minds perform even worse than humans in these aspects, and would die out.

In this universe, if one wishes to reach ones goals, one has to choose to (try to) do math/logic/deduction in the correct way; the way that delivers results. What works is determined by the laws of physics and logic that in our universe seem quite coherent and understandable (to a degree, at least).

There's no reason to be skeptical about whether I actually have some goals/preferences. And since I assume that I have some preferences, I have a need to conform to the correct way of doing math/logic/deduction, which is determined by what seems a rather coherent physical universe.

[James]

Subhan's question here, "How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally?" is such a gem.

I wonder if Eliezer intended it as parody.

[poke]

If somebody said to me "morality is just what we do." If they presented evidence that the whole apparatus of their moral philosophy was a coherent description of some subset of human psychology and sociology. Then that would be enough for me. It's just a description of a physical system. Human morality would be what human animals do. Moral responsibility wouldn't be problematic; moral responsibility could be as physical as gravity if it were psychologically and sociologically real. "I have a moral responsibility" would be akin to "I can lift 200 lbs." The brain is complicated, sure, but so are muscles and bones and motor control. That wouldn't make it a preference or a mere want either. That's probably where we're headed. But I don't think metaethics is the interesting problem. The deeper problem is, I think, the empirical one: Do humans really display this sort of morality?

[Unknown]

I've thought about Space Cannibals and the like before (i.e. creatures that kill one of the sexes during sexual reproduction). My suspicion is that even if such creatures evolved and survived, by the time they had a civilization, many would be saying to one another, "There really should be a better way..."

Evidence for this is the fact that even now, there are many human beings claiming it is wrong to kill other animals, despite the fact that humans evolved to kill and eat other animals. Likewise, in the ancestral environment, various tribes usually did kill each other rather than cooperate. But this didn't stop them from beginning to cooperate. So I suspect that Space Cannibals would do something similar. And in any case, I would fully admit that murder couldn't in fact be wrong for the Space Cannibals in the same way it is for us, even if there is an external moral truth.

In answer to Robin's question, assuming that morality exists, it probably has a number of purposes. And if one of the purposes is to preserve things in existence (i.e. moral truths correspond roughly with what is necessary to preserve things), then of course there will be a selection pressure to perceive moral truth. The disclaimer should not be needed, but this is not in any way a claim that it is moral to maximize inclusive genetic fitness.

[Richard]

Robin,
As Eliezer has pointed out, evolution is a nonhuman optimizer which is in many ways more powerful than the human mind. On the assumption that humans have a moral sense, I don't think we should expect to be able to understand why. That might simply be a problem which is too difficult for people to solve.
That aside, a man's virtues benefit the society he lives in; his inclination to punish sin will encourage others to act virtuously as well. If his society is a small tribe of his relatives, then even the weaker forms of kin selection theory can explain the benefit of knowledge of good and evil.

[Fiction Man]

Morality debates irritate me on so many levels. Treating everybody with respect seems to be a good solution for the moral reletivism debate.

[billswift]

Treating those who do not deserve respect with respect is basically spitting on those who do deserve it, especially those who work hard for it. I think you need to treat those you don't know with the "presumption of respect"; that is, if you don't know that they don't deserve it, assume they do. Borrowed from Smith's "presumption of rationality"; when you argue with someone, assume that they are rational until they demonstrate otherwise.

[Robin Hanson]

Richard, would you accept the same argument about God, that we know there is a God but don't really understand how we know, but gosh darn it we feel like there must be one so there must be one? Yes we evolved to help kin, and we expect many but hardly all other species to do this as well. But unless we know whether that behavior is moral we don't know if that is a process that makes our moral intuitions correlate with moral truth.

[Constant]

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

Let's detail the explanation for light to see if we can find a parallel explanation for morality. Brief explanation for light: light bounces off things in the environment in a way which can in principle be used to draw correct inferences about distant objects in the environment. Eventually, some animals evolve a mechanism for doing just this.

Let's attempt the same for morality. Brief explanation for morality: unlike light, evil is not a simple thing that comes in its own fundamental particles. It is more similar to illness. An alien looking at a human cell might not, from first principles, be able to tell whether the cell was healthy or sick - e.g. whether it has not, or has, fallen victim to an attack rewriting its genetic code. The alien may need to look at the wider context in order to draw a distinction between a healthy cell and an ill cell, and by extension, between a healthy human and an ill human. Nevertheless, illness is real and we are able to tell the difference between illness and health. We have at least two reasons for doing this: an illness might pass to us (if it is infectious), and if we select an ill partner for producing offspring we may produce no offspring.

Evil is more akin to illness than to light, and is even more akin to mental illness. Just to continue the case of mating, if we select a partner who is unusually capable of evil (as compared to the human average) then we may find ourselves dead, or harmed, or at odds with our neighbors who are victimized by our partner. If we select a business partner who is honest then we have an advantage over someone who selects a business partner who is dishonest. In order to tell apart an evil person from a good person we need to be able to distinguish an evil act from a good act.

This is only part of it, but there's a 400-word limit.

[Richard]

Robin,

Our moral intuitions correspond with moral truths for much the same reason that our rational predictions correspond with more concrete physical truths. A man who ignores reason will stick his hand back in the fire after being burned the first time. Such behavior will kill him, probably sooner rather than later. An man who is blind to good and evil may do quite well for himself, but a society whose citizens ignore virtue will suffer approximately the same fate as the twice-burned fool.

[Robin Hanson]

Richard, I agree that some social norms help a society prosper while others can "burn" it. And we have the intuition that morally right acts correspond to social norms that help societies prosper. But we would have had that intuition even if morally right acts had corresponded to the opposite. What evolutionary pressure could have produced the correct intuitions about this meta question?

[Z. M. Davis]

Constant, I would say that objective illness is just as problematic as objective morality; it's just less obviously problematic because in everyday contexts, we're more used to dealing with disputes about morality than about illness. You mention that "if we select an ill partner for producing offspring we may produce no offspring," and in an evolutionary context, probably we could give some fitness-based account of illness. However, this evolutionary concept of "illness" cannot be the ordinary meaning of the word, because no one actually cares about fitness.

I hate to use this example ("gender is the mind-killer," as we learned here so recently), but it's a classic one and a good one, so I'll just go ahead. Take homosexuality. It's often considered a mental disorder, but if someone is gay and happy being so, I would challenge (as evil, even) any attempt to define them as "ill" in anything more than the irrelevant evolutionary sense. I would indeed go much further and say that (for adults, at least) that which the patient desires in herself is health, and that which the patient does not desire in herself is sickness. (I actually seem to remember a similar viewpoint being advanced on The Distributed Republic a while back.) But this only puts us back at discussing preferences and morality.

[Dynamically Linked]

Constant wrote: So one place where one could critique your argument is in the bit that goes: "conditioned on X being the case, then our beliefs are independent of Y". The critique is that X may in fact be a consequence of Y, in which case X is itself not independent of Y.

Good point, my argument did leave that possibility open. But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

So if I were to draw a Bayesian net diagram, it would look it this:

      math ---   --- game theory ------------

              \ /                            \

                 --- evolutionary psychology - moral perceptions

              / \                            /

environment --   --- memetics ---------------

Ok, one could argue that each node in this diagram actually represents thousands of nodes in the real Bayesian net, and each edges is actually millions of edges. So perhaps the following could represent a simplification, for a suitable choice of "morality":

      math ---              - game theory ------------

              \            /                          \

               -- morality -- evolutionary psychology --- moral perceptions

              /            \                          /

environment --              - memetics ---------------

Before I go on, do you actually believe this to be the case?

[Caledonian]

I wonder if Eliezer intended it as parody.

He'd be making a serious mistake if so.

[denis bider]

Eliezer: You have perhaps already considered this, but I think it would be helpful to learn some lessons from E-Prime when discussing this topic. E-Prime is a subset of English that bans most varieties of the verb "to be".

I find sentences like "murder is wrong" particularly underspecified and confusing. Just what, exactly, is meant by "is", and "wrong"? It seems like agreeing on a definition for "murder" is the easy part.

It seems the ultimate confusion here is that we are talking about instrumental values (should I open the car door?) before agreeing on terminal values (am I going to the store?).

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

It is, however, much harder to talk about murders in general, and infeasible to discuss this unless we have agreed on a terminal value to work for.

[denis bider]

My earlier comment is not to imply that I think "maximization of human happiness" is the most preferred goal.

An easily obvious one, yes. But faulty; "human" is a severely underspecified term.

In fact, I think that putting in place a One True Global Goal would require ultimate knowledge about the nature of being, to which we do not have access currently.

Possibly, the best we can do is come up with plausible global goal that suits us for medium run, while we try to find out more.

That is, after all, what we have always done as human beings.

[TGGP]

Wanting to murder doesn't make it right. Nothing makes anything morally right.

[Richard]

Robin,

I don't understand your counterfactual.

"Good" and "Evil" are the names for what people perceive with their moral sense. I think we've agreed that this perception correlates to something universally observable (namely, social survival), so these labels are firmly anchored in the physical world. It looks to me like you're trying to assign these names to something else altogether (namely, something which does not correlate with human moral intuitions), and it's not clear to me how this makes sense.

[Robin Hanson]

Richard, if morality just meant social norms that help societies prosper, then of course we have little problem understanding how the two could be correlated, and how we could come to know about them. But if morality means something else, then we face the much harder question of how it is we could know about this something else.

[Dynamically Linked]

For those impatient to know where Eliezer is going with this series, it looks like he gaves us a sneak preview a little more than a year ago. The answer is morality-as-computation.

Eliezer, hope I didn't upset your plans by giving out the ending too early. When you do get to morality-as-computation, can you please explain what exactly is being computed by morality? You already told us what the outputs look like: "Killing is wrong" and "Flowers are beautiful", but what are the inputs?

[Fly]

EY: "human cognitive psychology has not had time to change evolutionarily over that period"

Under selective pressures, human populations can and have significantly changed in less than two thousand years. Various behavioral traits are highly heritable. Genghis Khan spread his behavioral genotype throughout Asia. (For this discussion this is a nitpick but I dislike seeing false memes spread.)

re: FAI and morality

From my perspective morality is a collection of rules that make cooperative behavior beneficial. There are some rules that should apply to any entities that compete for resources or can cooperate for mutual benefit. There are some rules that improved fitness in our animal predecessors and have become embedded in the brain structure of the typical human. There are some rules that are culture specific and change rapidly as the environment changes. (When your own children are likely to die of starvation, your society is much less concerned about children starving in distant lands. Much of modern Western morality is an outcome of the present wealth and security of Western nations.)

As a start I suggest that a FAI should first discover those three types of rules, including how the rules vary among different animals and different cultures. (This would be an ongoing analysis that would evolve as the FAI capabilities increased.) For cultural rules, the FAI would look for a subset of rules that permit different cultures to interact and prosper. Rules such as kill all strangers would be discarded. Rules such as forgive all trespasses would be discarded as they don't permit defense against aggressive memes. A modified form of tit-for-tat might emerge. Some punishment, some forgiveness, recognition that bad events happen with no one to blame, some allowance for misunderstandings, some allowance for penance or regret, some tolerance for diversity. Another good rule might be to provide everyone with a potential path to a better existence, i.e., use carrots as well as sticks. Look for a consistent set of cultural rules that furthers happiness, diversity, sustainability, growth, and increased prosperity. Look for rules that are robust, i.e., give acceptable results under a variety of societal environments.

A similar analysis of animal morality would produce another set of rules. As would an analysis of rules for transactions between any entities. The FAI would then use a weighted sum of the three types of moral rules. The weights would change as society changed, i.e., when most of society consists of humans then human culture rules would be given the greatest weight. The FAI would plan for future changes in society by choosing rules that permit a smooth transition from a human centered society to an enhanced human plus AI society and then finally to an AI with human origins future.

Humans might only understand the rules that applied to humans. The FAI would enforce a different subset of rules for non-human biological entities and another subset for AI's. Other rules would guide interactions between different types of entities. (My mental model is of a body made up of cells, each expressing proteins in a manner appropriate for the specific tissue while contributing to and benefitting from the complete animal system. Rules for each specific cell type and rules for cells interacting.)

The transition shouldn't feel too bad to the citizens at any stage and the FAI wouldn't be locked into an outdated morality. We might not recognize or like our children but at least we wouldn't feel our throats being cut.

[Richard]

Robin,

I don't know how people are capable of discerning moral truths. I also don't know how people are capable of discerning scientific or mathematical truths. It seems to me that these are similar capabilities, and the one is no more suprising or unlikely than the other.

[Robin Hanson]

Richard, while there are surely many details we would like to understand better, surely we understand the basic outline of how we discern scientific and mathematical truths. For example, in math we use contradiction to eliminate possible implications of axiom sets, and in science we use empirical results to eliminate possible abstract theories. We have nothing remotely similar in morals. You never said whether you approved of a similar argument about knowledge of God.

[Constant]

Z. M. Davis writes: ... objective illness is just as problematic as objective morality

I would argue that to answer Robin's challenge is not necessarily to assert that there is such a thing as objective illness.

Accounts have been given of the pressure producing the ability to see beauty (google sexual selection or see e.g. this). This does not require that there is some eternal beauty written in the fabric of the universe - it may be, for example, that each species has evolved its own standard of beauty, and that selection is operating on both sides, i.e., selecting against individuals who are insufficiently beautiful and also selecting against admirers who differ too far from the norm.

However, this evolutionary concept of "illness" cannot be the ordinary meaning of the word, because no one actually cares about fitness.

My argument is: people can distinguish illness because it enhances their fitness to do so. Compare this to the following argument: people can distinguish the opposite sex because it enhances their fitness to do so. Now, okay, suppose that people don't care about fitness, as you say. Nevertheless, unbeknownst to them, telling women apart from men enhances their fitness. Similarly for illness.

Take homosexuality. It's often considered a mental disorder, but if someone is gay and happy being so, I would challenge (as evil, even) any attempt to define them as "ill" in anything more than the irrelevant evolutionary sense.

Homosexuality reduces fitness (so you seem to to agree), but this does not make it an illness. Not everything that reduces fitness is an illness. Rather, illness tends to reduce fitness. Let me put it this way. Blindness tends to reduce fitness. But not everything that reduces fitness is blindness. Similarly, illness tends to reduce fitness. But that doesn't mean that everything that reduces fitness is illness.

... that which the patient desires in herself is health, and that which the patient does not desire in herself is sickness.

We can similarly say, that which a person desires in a mate is beauty. However, I think the most that can be said for this is that it is one concept of beauty. It is not the only concept. The idea that there is a shared standard of beauty is, despite much thought and argument to the contrary, still with us, and not illegitimate.

[Richard C]

"what possible pressure could produce an ability to see morality?"

Unlike the other Richard, I don't think we "see" morality with a special "sense", or anything like that. But if we instead understand morality as a rational idealization, building on our perfectly ordinary general capacity for systematizing judgments so as to increase their overall coherence (treating like cases alike, etc.), then there's no great mystery here.

[Constant]

Dynamically Linked writes: But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

According to Tegmark "there is only mathematics; that is all that exists". Suppose he is right. Then moral truths, if there are any, are (along with all other truths) mathematical truths. Unless you presuppose that moral truths cannot be mathematical truths then you have not ruled out moral truths when you say that so-and-so is not contingent on anything except mathematics and such-and-such. For my part I fail to see why moral truths could not be mathematical truths.

Before I go on, do you actually believe this [Bayesian net diagram] to be the case?

I'm sorry to say that I can't read Bayesian net diagrams. Hopefully I answered your question anyway.

[Richard]

Robin:

Discarding false mathematical and scientific conjectures is indeed much easier than discarding false moral conjectures. However, as Eliezer pointed out in an earlier post, a scientist who can come up with a hypothesis that has a 10% chance of being true has already gone most of the way from ignorance to knowledge. I would argue that hypothesis generation is a poorly-understood nonrational process in all three cases. A mathematician who believes he has found truth can undertake the further steps of writing a formal proof and submitting his work to public review, greatly improving his reliability. A man confronted with a moral dilemma must make a decision and move on.

I think that the universal tendency towards religion is indeed evidence in favor of the existence of God, but not very strong evidence. The adaptive advantage of discerning correct metaphysics was minimal in the ancestral environment.

Richard C:

I think if you try to use your "general capacity for systematizing judgments" to make moral decisions, you'll restrict yourself to moral systems which are fully accessible to human reason.

[Richard Hollerith]

If I were leading this discussion instead of Eliezer, a few days ago I would have pointed out that although a human and an AGI both have a system of terminal values, a.k.a., a goal system, it is easier to understand and to get right the the goal system of an AGI than of a human.

One reason for that is that an AGI will probably have the transparency property. A second reason values-for-humans is harder than values-for-AGI is that a well-designed AGI will be vastly less biased than the least-biased human beings. It seems to me that a large part of the reason human ethical codes are the way they are is the need to try to neutralize and
counteract common human biases.

Then I would have told the audience that although comments about terminal values for humans is not completely off-topic, I am particularly interested in the problem of how a creator of an AGI picks the goal system of the AGI.

Then I would have explained monolithic AGI: it might be the case that the future light cone will be inhabited by a single AGI rather than a "society" of individuals. Although this monolithic AGI might decide to create a vehicle and send it to the moon, that lunar vehicle will with extremely high probability never compete with its creator (because the monolithic entity is extremely good at designing smart lunar vehicles and other intelligent artifacts that do not come to have their own agendas).

I would have done that just to spare myself from reading many comments to effect that the purpose of morality is to enable individuals to live and to work together effectively. That is a useful observation for explaining how human morality got the way it is, but IMHO not useful for the difficult problem of choosing a goal system for an AGI that might become smarter than us.