« Complexity and Intelligence | Main | Today's Inspirational Tale »

November 04, 2008



Ever read Zen and the Art of Motorcycle Maintenance? I couldn't help but think of Pirsig when I read that we should "take no pleasure when you and your friends disagree with others; that is the road to rationality ruin."

Unless I've missed some subtle irony or sarcasm in your last paragraph, it would appear that you're calling for massive group think and to flee in terror from any thoughts other than what the majority think.

Evil? Sorry, maybe I'm obtuse but you lost me on this one.

Of course that's not what he's saying. He's saying you should take no pleasure in the opinions of others, whether they are like yours or not. You should take pleasure in being right.


"group think"

Robin is calling for the opposite! Isn't it when we cluster in smug cabals cackling that we alone have the truth, happily signaling our loyalty to each other and re-inforcing that loyalty, that we are trapped in an ugly echo chamber where real thought is replaced by dogma?

If we want to talk about Plato, isn't the key dialogue here Meno? "By Zeus, I may no longer know. . ."

Learn to see it in yourself, and then learn to be horrified by how thoroughly it can poison your mind.

How do you recommend not enjoying something that you actually enjoy? Are some methods better in the long run than others?

I can't get to the link. If it's broken, can you fix it?

How do you recommend not enjoying something that you actually enjoy?

Religions have worked on this problem for thousands of years.

"We feel a deep pleasure from realizing that we believe something in common with our friends, and different from most people. We feel an even deeper pleasure letting everyone know of this fact. This feeling is EVIL."

- haha. This post evokes exactly that feeling in me: isn't it nice that in this world full of irrational retards, there are still some truly sensible, rational people around, and that we can gather here on O/B? Especially when the irrational retards coming under fire are also RELIGIOUS.

Nancy Lebowitz: How do you recommend not enjoying something that you actually enjoy? Are some methods better in the long run than others?

By recognising that it is poison and the enjoyment is trash. How tempted would you be to use lead acetate as a sugar-free sweetener?

Although, it really does say something sad about our world that a holder of the most reasonable beliefs imaginable [non-existence of God, likelihood of smarter than human intelligence at some point in the next 100 years and the idea that this would be quite an important development] finds himself in a tiny minority...

this is exactly the kind of post which i will miss dearly if/when robin stops blogging here. pure awesomeness robin.

"""Although, it really does say something sad about our world that a holder of the most reasonable beliefs imaginable finds himself in a tiny minority..."""

It's a lot worse than this. He doesn't exist, and might never exist.

Roko: "This post evokes exactly that feeling in me: isn't it nice that in this world full of irrational retards, there are still some truly sensible, rational people around, and that we can gather here on O/B?"

I was doing this too, the other night. Scary, isn't it?

Yes, increasing coherence with decreasing context is inherently degenerate.

Yes, these posts by Robin are valuable, to the extent they don't impair his wider scope of effectiveness.

Yes, to Nancy Lebovitz that some methods are more effective than others, but we've evolved to view methods as if with closure, and enjoyment as if primary, whereas it's increasingly important that we take a broader systems view of our place as agents acting within an environment of increasing uncertainty, such that all methods have unforeseeable consequences, and 'tis better to support ongoing growth supporting ongoing enjoyment, than to support enjoyment as presently perceived.

Yes, recursion is popularly non-intuitive, but for profoundly information-theoretic reasons, vital to growth.

Yes, it's sad that leading-edge thinking is lonely, and natural such gradients are to be welcomed. If you would simply have the leading-edge become the norm, then please recurse to the beginning of this comment.


I can accept that a tendency could be dangerous or even necessarily harmful. But 'evil' is not a concept that can be applied.

Much as I'd like to agree with you on instinct, my best rational assessment is that group-think is morally neutral. Most humans lack the coherent rational processes necessary to error check themselves. Consensus, voting, and so on are useful tools for error checking: NASA uses such a technique (multiple separate flight computers, some with independently written code) to guard against unforeseen systemic errors.

Like any immune system within a larger organism, its vulnerable to subterfuge and subversion. It doesn't follow, however, that the underlying process is evil: flawed and outdated, perhaps, but it still fulfills many of the needs it arose in reply to. Any rational system which proposes to do away with group-think had better find something which performs its job at least as well in practice.

Perhaps letting everyone else know of one's supposedly superior viewpoint is a form of verbal status signaling. It may also be a working instance of a mechanism that generates pleasant, positive cognitive illusions that foster social cohesion.

Robin's choice of terminology here is an interesting one. Etymologically speaking, "evil" is related to the German "uebel" (sorry, no umlauts on this keyboard). Take a look at the word "ill," and see that it is also related to the words evil and uebel. The opposite of what is pleasurable is that which is disgusting, illustrated in part, by the etymology. In this instance you have to use your rational prefrontal cortex to override the impulse to experience pleasure and replace it with a thought of disgust. This is a rather Spinozan, which is to say, difficult and rare thing to which to aspire. I predict that those who approach the closest will be among the minority.

Emotion becomes evil when it results in things you don't approve of. This particular emotion might be dangerous by default, but I think one can learn to control it, so that on the balance it starts to push on the positive side of things, motivating rather then driving insane. So, it might be a good idea to avoid placing this kind of emotional weight on your beliefs until you are sufficiently sure of them, and to check their rational strength from time to time, but this emotion doesn't look so lethal as to warrant destroying it by unconditional injunction.

Vladimir, beware of thinking you can control something this powerful. Think of the rings of power from Lord of the Rings.

I've got to side with Hanson on this one. The pleasure of believing what is true is distinct from the pleasure of believing a strange private truth along with your close friends. It's pretty hard to see the latter as a force for good.

Robin, but you need to draw the line somewhere, to decide whether each particular emotion is dangerous in particular contexts, otherwise it is a road to the fake Vulcan morality. One needs to leave room for taking pleasure in real, even though no belief has probability 1, and allowing more emotions may hurt your chances.

Also, controlling emotions by training yourself to not feel them doesn't always work, sometimes to be successful you need to make injunctions against placing yourself in contexts that activate them, which in cases like this may cripple you beyond hope. There are all sorts of negative consequences, and sometimes positive effect is minor, so it's really a tradeoff, not obvious decision.

So, I shouldn't feel good that me and my associates have decided that the God of Abraham doesn't exist?

Eliezer, but what about the case when strange private truth is actually real, which also falls under Robin's injunction. Blocking emotions in that case would be a mistake, so it all turns to not regarding this instrumental action as end in itself, and deciding on the merits of each particular case. It might turn out that corruption is deep beyond repair, but it isn't at all obvious for me that this is necessarily the case here.

- haha. This post evokes exactly that feeling in me: isn't it nice that in this world full of irrational retards, there are still some truly sensible, rational people around, and that we can gather here on O/B? Especially when the irrational retards coming under fire are also RELIGIOUS.

Thank you for providing such an excellent demonstration of the typical rationalist's intellectual maturity.

Really, "retards"? Given your supposed mental superiority one would think you could have come up with an insult that wouldn't make you sound like a middle-schooler.

Eliezer - where is the gain or contribution in knowing a commonly-held truth? Whether you view it as getting an advantage or contributing to society, it is the strange private truths which advance common knowledge. A trader who knows just what the market knows knows nothing. Same goes for a scientist.

I see this bias as counteracting the bias of groupthink. The opposite bias is for people to enjoy believing what everyone else believes. This leads to homogeneity of viewpoints, less generation and testing of new hypothesis, and stasis. The people who enjoy believing they have a secret truth are those who nurture non-mainstream but plausible hypotheses, and accumulate new evidence to possibly challenge the mainstream. I think this is very valuable.

A recent case is carbs vs. fat - the mainstream believed (on very shoddy evidence) that fat was bad and carbs were good. A minority of people enjoyed eating high fat low carb diets, flaunting the fact, and flaunting the results (good blood lipid levels, not gaining wait). They sustained an alternative hypothesis. Now, scientific consensus is catching up to the actual evidence.

It can be dangerous to one's status, reputation, and career to challenge commonly held truths. I'm very glad that people enjoy doing it, as a counterbalance to that danger.

Patri, I'm not sure there is clear evidence of a "groupthink" bias. Maybe I should do a post on that.

Z.M Davis: "I was doing this too, the other night. Scary, isn't it?" - well, perhaps you could call it scary. But it binds us together...

Anti-Reductionist: "Thank you for providing such an excellent demonstration of the typical rationalist's intellectual maturity." - I was trying get a bit of self-effacing humor in there...

You know, perhaps we are seeing this article the wrong way. Perhaps we are rather like pacifists who refuse to use weapons because "war is evil".

If we make effective meme-spreading techniques "evil" then only "evil" memes will spread effectively.

Perhaps we could see this article as support for the idea that we need a rational religion, somewhat like the Cosmic Engineers.

Robin Hanson,
maybe there's a tremendous miscommunication going on, but I think that there's a groupthink bias and I think that most people who read this blog think so. If you agree with my assessment of the audience, then you should definitely write about it!

(I suppose this post indicates a reason why we would fabricate a groupthink bias.)


There is plenty of research under the label of "group polarization," "conformity effects," "social desirability bias," etc. What does 'groupthink bias' mean to you?

Roko: "- well, perhaps you could call it scary. But it binds us together..."

I hope you're joking.

I think we need to distinguish between two sorts of "groupthink".

One is the rational kind, where a person sides with a group because doing so is a decent heuristic. If I know nothing about finance, I may be inclined to believe people I think know more than me (Ben Bernanke, the media, etc) when they say we are in a financial crises. I don't see anything irrational or evil here. If everyone is running away from a building you're in screaming about a bomb, is it irrational to follow them?

What Robin seems to be talking about is a desire to side with a specific group over another because it signals my devotion to that group. If I am a liberal, I may be inclined to believe there is a financial crises because it makes conservatives look bad. I would say this is evil, because acting on these desires almost always obscures the truth.

The most successful societies seem to have ways of signaling loyalty to social norms without polluting the truth of things. For example, we no longer have to be loyal to a specific politician (or monarch) or a specific religion in order to be seen as "part of the group" and thus worthy of trust.

The example given is peculiar in that it allows rationalists to collectively scoff at those religious types throughout history who created gods in their own image. This may have actually created a comments thread version of circling the wagons against such irrationality; 'Oh no, we would never believe such a thing. Those who do are not as wise as we, banding together in such shared understanding as we do.'

Let's try expanding the principle to prick the conscience of all. People project their own ideals and positions onto a politician (who indeed depends on being a sufficiently blank slate to fit the purpose). We might blithely adopt assumptions about people we date, only to discover that, months or years later, Sally does not like children or John does not appreciate the benefits of suburbia.

I don't think that this is a conscious or deliberate ploy, but rather a tacit desire to experience camaraderie.

good point well made, Robin.

This reminds me of the so-called "Galileo Gambit", the cliche that someone who disagrees with the majority opinion on a given question, is therefore virtuous and probably right. I'm sure that you can think of plenty of examples of people with unpopular ideas who wear their status as a "heretic" as a positive badge of honor. They're prominent amongst global warming skeptics, creationists, historical revisionists, etc.

I think the "Evil Pleasure" that Robin alludes to is part of this, but there's also the fact that the Western intellectual tradition encourages us to question everything, "think for ourselves" and "take no-one's word for it". That goes back to Socrates. Unfortunately it's easy to take it too far.

I write about this here

And by corrollary, take no pleasure in agreement with a majority for its own sake. Correct?

I enjoyed your post, Robin.

It'll keep me coming back for more and maybe I'll become smarter, more rational and less biased.

You say it's evil to take pleasure in certain beliefs. I say it's nonsensical to take pleasure in any belief. While many activities may be pleasurable (swimming, eating, movie-watching, scientific experimentation, etc.), it's a stretch to include "believing" in that number. A belief is but a temporary locus of stability, a local maximum, for an inquisitive mind. I enjoy swimming, and look, I swam all the way out to that island! I enjoy eating, and look, I ate that whole pie! One would not renounce swimming or eating after a particularly satisfying result in either activity. And yet beliefs seem to be stationary markers: here is where my mind stopped! Fun!

One may use her recent results in any activity as a status symbol or whatever, but results are secondary to the activity itself, and somewhat contingent anyway. If one's beliefs give her pleasure, the source of the pleasure is not any particular belief but rather her status-seeking and whatever other activities her belief enables. Even though success in, for example, swimming might similarly enable status-seeking etc., swimming is actually a potentially pleasurable activity in and of itself. Believing is not. Thinking may be a pleasurable activity, but these things are not the same. If one finds one's beliefs to be beautiful, then perhaps contemplation of belief might be pleasurable, but that's rather far off the track. I know several thoughtful people who profess traditional religion, and I think even they would deny pleasure in belief. Thoughtless people of course take pleasure in any number of questionable things.

I try to avoid beliefs as such. I have instead habits of mind, which I try to change on a regular basis. I try not to let "truth" distract me from method and practice. Evil is problematic in the mental context, in any case. Let's reserve this word for actions in the corporeal realm.

Roko, don't be defensive. Your comment was the thread-winner; the haters are just tone-deaf.

Robin: Is it evil because it swamps the "getting the facts right" signal with a "I'm a member of the in-group" signal? (Thus inclining you to spend less energy getting the facts right, and more demonstrating your membership of the in-group with loud protestations of your belief in their stated ideas?) Or is there some other aspect of this that I'm missing?

Nancy: I think guidance about emotions being evil is pretty hard to use (this also comes up in Christian teachings, with Jesus telling people more-or-less not to get angry or lustful). What is useful for me is to see them as signposts to a bad place. For example, when I'm feeling a certain kind of self-satisfied righteous anger, I know I'm about to say or (more commonly) post something that I will later regret. It's not that the feeling is evil--the feeling's morally neutral. But that feeling is also a signal that I'm likely to be doing a bad job of thinking through what I say. Similarly, it's really important to recognize that when you're scared, angry, horny, embarrassed, etc., those feelings have an impact on how you think, and impose a certain kind of bias on your perception of reality.

I always found sports fans annoying. Now I am part of an in-group that considers them evil! Thanks Robin!

Robin is basically cautioning against every dinner party i have ever been to. And he's right!

This kind of argumentation is circular, and commits the classic logical fallacy of trying to argue away absolute (religious) truth:

- Whenever a group of people think they have "the truth" they become biased and evil
- Many religious groups (ex. Evangelical Christians) often claim to have "the truth"
- Therefore these religious groups are evil
- Let's form our own group, based on our declaration of knowing that all other groups that espouse "the truth" are misguided
- Our group has "the truth" about "the truth"
- Oh wait, that makes us evil, too!

See how ridiculous this gets?

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30