« Open Thread | Main | OB meetup Friday 13th 7PM Springfield VA »

March 01, 2009

Comments

My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you'll make in the future, such as Newcomb's Problem, Parfit's Hitchhiker and of course the Prisoner's Dilemma. It explicitly doesn't handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.

We could try to consider fully arbitrary second-order rationality as the algorithm adopted by a first-order maximizing agent under these conditions.

But then we can always have the God who only rewards you for starting with an algorithm that outputs the first decision in alphabetical order, and who will not reward you for self-modifying to an alphabetic algorithm unless you do so because "alphabetizing" was the first algorithm in alphabetical order that you considered, and who will not reward you if you self-modify "in order to maximize utility".

So I explicitly restrict my decision theory to the realms of "other people care about the decisions they predict my algorithm will make under what conditions, but only care about these decisions and don't care what algorithm makes it apart from that" because this is the problem that turns out to have an elegant first-order solution that doesn't require precommitment or further self-modification. I'm not aware of any elegant first-order solution to a decision problem more general than this, and so any more general decision problems would have to be handled by an application of the first-order theory to maximize on a second-order choice between algorithms.

I wouldn't call it faith in rationality; it's more like a definition. "Rationality" just means the habits of thought that result in win. A peculiar consequence of this definition is that the meaning of rationality depends on the universe outside your mind. In particular if there are gods or aliens that can read and understand your internal state, then the results can be counterintuitive.

Only Box B seems win-win to me. either I get a million dollars or I proved the predictor wrong (which should be good enough to get me on talk shows and a book deal).

So I explicitly restrict my decision theory to the realms of "other people care about the decisions they predict my algorithm will make under what conditions, but only care about these decisions and don't care what algorithm makes it apart from that"
(I think that positing that the Predictor is infallible, and yet doesn't know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can't happen, so you can't ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)
My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you'll make in the future, such as Newcomb's Problem, Parfit's Hitchhiker and of course the Prisoner's Dilemma. It explicitly doesn't handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.
Sure. Hence my disclaimer that my paradox relies on God caring about your motives / source code.

But I think you have to admit that we could all be sims in the computer of a sadistic God. And then you can no longer believe that rationality is always the best strategy. You can at best find some condition which guarantees that rationality is the best policy.

(If the answer is that black-box rationality, in which the algorithms used by agents are unknowable to anyone, is such a condition, that would be interesting. It wouldn't apply to the real world; and I don't think that Newcomb's paradox could exist in such a world - the required superintelligence could not exist, as noted above. So I think the fact that you're interested in Newcomb's paradox implies you would rather have a more general condition. For instance, the stipulation that any knowledge about an agent's internals is public knowledge, or is possessed by that agent.)

I'm still curious how your solution works without requiring precommtment.

If you start with flawed premises, anything you build upon it will be shaky. An inconsistent system of thought like religious faith will generate cases of action where there is no right thing to do. You can expect idiocy to propagate to all levels of something that's been built upon a basis laid with one or more nonsensical assumptions.

"Rationality" just means the habits of thought that result in win.
It doesn't. Rationality must have something to do with how you think. Otherwise you should use the term "winning". It doesn't appear to me that rationality is always the best policy for humans. (It would certainly be a more competitive strategy if we had no subconscious component to our thinking.) People with overly-optimistic self-evaluations are happier and have better outcomes. In many cultures, rationality would be certain to make you unhappy and likely to get you killed.

"Rationalism" might, by analogy to other "isms", mean the dogmatic belief that rational strategies are always the best strategies. A lot of people seem to believe that's what it means. If that's what it means, then I'm not a rationalist.

Philip Goetz: Wouldn't that depend on what each person means by "winning". If your only value and goal in life is to "be happier and have better outcomes", then the rational thing to do might be to "have overly-optimistic self-evaluations" (if there really was nothing that would make them even happier).

No, Mr. Goetz, you are misinformed. I don't think there is any Christian church, conservative or otherwise, that teaches the doctrine you describe.

Traditional Catholic doctrine was that salvation depends on the grace of God, mediated through the offices of the church. Final thoughts on sinfulness would increase the time of one's penance in purgatory. But lately they have been soft-pedaling the purgatory business Evangelical christians believe that one moment of sincere belief in Jesus secures salvation. If you have an honest born again experience, then any subsequent sins do absolutely nothing. Calvinists and their progeny (e.g. Presbyterians) believe that one is predestined for damnation or salvation, so last minute thoughts are likewise insignificant. If there is a church you know of that teaches this, please provide an example and a quotation from their books of doctrine.

Augustine did write about a related problem, namely that one might game the system of righeousness by planning to sin then confess. Hence his description of praying as a young man with a mistress, "God, grant me chastity, but not yet." You can find that in the Confessions, I don't have it to hand, but it is in chapter 3 or 4. But I don't think he discusses salvation in that regard.

But there are popular literature examples of what you are talking about. In Neal Stephenson's Quicksilver the Shaftoe boys use that theory to sell their services expediting hangings to condemned criminals. Is that book where you are getting your Christian theology? It is very good, but not as a guide to theology.

More generally, on whether rationality wins by definition, you guys are wandering in the dark. Read Thomas Schelling's Strategy of Conflict, chapter 2. There are MANY strategic situations in which rational agents badly underperform irrational ones. Two irrational people -- who have honor or loyalty or a belief in a god who watches them -- collectively outperform two irrational people in the Prisoner's Dilemma.

Moreover, irrationally vengeful agents outperform rational ones as recipients in the Ultimatum Game. See Guth, Schmitberger and Schwarze, 1982. Or there is a good review in Camerer's Experimental Game Theory.

Further, irrational males who feel love will tend to outperform rational ones in securing the reproductive labor of discriminating females.

Irrationality usually outcompetes rationality. That is why there is so much of it. The reason you have emotions is that they evolved to help you outperform rational agents in game theoretic dilemmas. Try reading Robert Frank's passions within Reason for a good introduction to game theory, evolution and winning strategies based on irrationality.

A smart, rational person will choose to be irrational at many times. I think cultivating the beneficial types of irrationality is a much smarter strategy than Eliezer's irrational love of rationality despite the consequences.

And that includes cultivating religion. Religious people live longer, report far greater subjective happiness, and have more sex than atheists. They also support charities, volunteer, and give blood at much higher rates. That's why religion is universal among human societies, because it grants such a massive competitive advantage, both in individual game theoretic situations and in fostering social solidarity.

Why do you guys think it is everywhere? Don't you realize that societies compete? If atheism gave a society advantages, there would, you know, be a history of atheist societies actually existing for more than, what, 60 years is the current max? And the soviet union didn't end so well. Modern Europe is a good example of a pretty successful society that is at least effectively atheist. But it is unable to defend itself, relying on religious americans in the military to defend it, and unable to reproduce itself. Maybe a long-term successful atheist society could exist, but so far it hasn't. That should give you guys pause, at least if you are, you know, rationally updating your beliefs against the historical data.

(I think that positing that the Predictor is infallible, and yet doesn't know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can't happen, so you can't ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)

I'd say that positing that the Predictor is infallible, and yet doesn't know what algorithm you use is a lot more like positing an oracle machine than like asking what GR predicts upon the sudden vanishing of the Sun. Or more briefly: this gedankenexperiment is allowed.

It doesn't [just means the habits of thought that result in win]. Rationality must have something to do with how you think.

How about "the optimal use of information to choose a goal-directed action". Implication: acting rationally will win if it's at all possible.

I think that positing that the Predictor is infallible, and yet doesn't know what algorithm you use...

Eliezer said that the Predictor didn't care what algorithm you used, not that it didn't know. That is, the Predictor predicts your algorithm, but it puts money in Box B or not depending only on what you decide. That's how Newcomb's Paradox differs from Augustine's Paradox (and Kavka's Toxin Puzzle, for another example).

If you start with flawed premises, anything you build upon it will be shaky. An inconsistent system of thought like religious faith will generate cases of action where there is no right thing to do. You can expect idiocy to propagate to all levels of something that's been built upon a basis laid with one or more nonsensical assumptions.
The Predictor of Newcomb's paradox is less likely than a God. So why is it legitimate to ask about Newcomb's paradox, but not legitimate to ask about God?
No, Mr. Goetz, you are misinformed. I don't think there is any Christian church, conservative or otherwise, that teaches the doctrine you describe.
The second part, that you go to heaven if you sincerely repent (and meet other conditions that may include confession, baptism, etc.) before dying, is taught by almost every Christian church, conservative, Catholic, Protestant, or otherwise. (The major exception is forms of predestination that teach that you can never know whether you are saved or not.) That is the only part that matters for this discussion.

(Catholic believe that you can go to purgatory even if you die immediately after confession. Therefore, it wouldn't make any sense for them to place as much importance on confession as they do if they did not believe that dying unshriven condemned you to Hell. The Catholic teaching that babies who die go to Limbo, which was only retracted 2 years ago, would also make no sense. The Catholic position that suicides go to Hell, which is expertly dodged nowadays if you ask about it but is nonetheless the dominant Catholic belief on the subject, would also make no sense. Furthermore, you can find many explicit descriptions of this belief; the most famous is in Hamlet. Regardless of what the Catholic Church teaches, this is what most Catholics throughout history have believed. But, again, it has no bearing on my post either way. Many protestants believe that if you're saved once, you remain saved. Doesn't matter WRT my post.)

Agree about your observations that rationality doesn't always win. That was the point of my original post.

Communist states are not atheistic. Communism resembles Christianity and Islam more than Judaism does.

Philip Goetz: Wouldn't that depend on what each person means by "winning".
Yes. Definition of "winning" is unrelated to the point I was replying to.

Would someone be so kind as to define "rationality" in the context of this discussion? Many of the comments so far seem to be working from a definition that I don't think matches the one I would use by default.

For instance, in what way is it reasonable to discuss "irrational" behaviors outperforming "rational" behaviors? Wouldn't it be, by definition, not rational to persist in behaviors that are demonstrably suboptimal?

Only Box B seems win-win to me. either I get a million dollars or I proved the predictor wrong (which should be good enough to get me on talk shows and a book deal).
I did it again. I matched your response to my set of expected misunderstandings based on surface similarity, and initially dismissed it, before realizing what you'd actually said.

Clever. :)

Patrick wrote:

Eliezer said that the Predictor didn't care what algorithm you used, not that it didn't know. That is, the Predictor predicts your algorithm, but it puts money in Box B or not depending only on what you decide. That's how Newcomb's Paradox differs from Augustine's Paradox (and Kavka's Toxin Puzzle, for another example).
Oops! You're right. Nonetheless, my main point still stands, which is that it is possible that rationality doesn't always win.

frelkins: Sorry for my sloppiness. The only thing that matters for my original post is that it is allowable to sin all your life long, and repent in your final moments. This is believed by all Christian sects that I know of, with the exception I noted on predestination, probably because of the thief on the cross, who repented at the last minute and was saved, which I mentioned above. There is disagreement on what caused you to repent (e.g., God or free will), but that's irrelevant.

(I hold to my previous comment that Catholics believe that you are damned if you die without confessing. Maybe the Pope doesn't believe it; maybe a lot of Catholics don't believe it today; but over the past 1000 years, the evidence I know of convinces me that most Catholics believed it. I said "babies" who die go to limbo; I meant unborn fetuses. That was the official Catholic doctrine until 2 years ago. Either way, as I've already noted, that part has no impact on my arguments.)


Religious people live longer, report far greater subjective happiness, and have more sex than atheists. They also support charities, volunteer, and give blood at much higher rates.

Several problems here. First, self-reported happiness does not sound like a reliable indicator of actual happiness. And the statistics for some things mentioned vary by the poll. One poll conducted by christians, for example, found that atheist doctors volunteer more than theist ones. And as to more sex. That's not necessarily an indicator of better living. For example, unmarried people typically have less sex than married ones. If theists marry younger than atheists then this alone will probably put them into the category of having more sex on average. But its hardly necessarily a good thing to get married early.

And then there are the clearly negative numbers. Believers are more likely than atheists, according to many articles I've read, to get pregnant outside of marriage, to have abortions, to be child abusers and molesters and to end up in prison.

The personal advantages of religion are far from as clear as you've characterized them.


"Rationality" just means the habits of thought that result in win.

Not as I define the term "rational". They might result in winning more often than not (depending on the variety of win in question). But clearly being rational can also be a detriment---especially in a society where irrationality is exalted and rewarded---which is, to one degree or another, every society I'm aware of.

Philip Goetz: Wouldn't that depend on what each person means by "winning". If your only value and goal in life is to "be happier and have better outcomes", then the rational thing to do might be to "have overly-optimistic self-evaluations" (if there really was nothing that would make them even happier).

This.

Many times the "rational" thing to do isn't the rational thing to do isn't the rational thing to do.

Some research shows that those of us who do lots of research into potential product purchases are less happy with their purchase than those who don't. Thus, the rationally irrational thing to do is not obsess over doing such research.


The optimal strategy is to be selfish all your life, without intending to repent, and then repent in your final moments and truly mean it.

When the stakes are either eternal joy or eternal agony, it seems to me that the optimal strategy is to be unselfish all your life.

Even if there were no afterlife I would think a life strongly motivated by kindness would still be an "optimal strategy". It seems to me an intrinsically better life in its own right than that of an egoist. At the least, I'm not convinced the contrary is true.


Some research shows that those of us who do lots of research into potential product purchases are less happy with their purchase than those who don't. Thus, the rationally irrational thing to do is not obsess over doing such research.

Perhaps we should simply conclude that rationality and happiness do not necessarily overlap as much as some would prefer to think.

Which is fine with me since, while happiness is intrinsically good, its not, to my mind, the only intrinsic good.

Phil Goetz:
The doctrine that you go to heaven if you repent is not the full extent of your claim. You also claimed that "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life." False. Try and produce some support for modern churches teaching that. You will fail.

Frelkins. I agree with almost all. But I think the stuff about Mary being co-redeemer is the view of a small minority (lead by a group of Italian ladies in Bayside, Queens. Rejected by the official church doctrinal office during the late years of the last pope, iirc.

Soulless (love the handle). Eliezer might have his own definition of rational. I am working with the standard game theoretic version which assumes that rational = 1) self interested (ie no social utility, you care about yourself but not others), 2) value-maximizer (that you always want higher payoffs). The definition gets squishy right away once you try to define payoffs. e.g. What is the difference between a rational being with a payoff on receipts to others as part of his utility function and an irrational altruist? The literature goes on and on about that without a resolution. Then there is a big literature defining ways of being irrational. You would like anything Kahneman and Tversky wrote about that, or Max Bazerman has a good basic summary called, um, Judgment in Managerial Decision Making (business schools are the largest audience for this kind of thing).

But I think it is pretty standard to say that emotions are not rational, e.g. you get mad at someone and chose conflict even when the payoff to surrender are higher. Or you love your wife and stay faithful to her even when there is someone younger and prettier, cause you would feel guilty if you left. In each case, people choose lower payoffs because of a bias in their cognitions. But you can define their payoff structure so that that isn't irrational, but then once you take that step there is no way to define anything as irrational. Just people rationally following their strange utility structure.

But it is I think pretty well accepted that no human being are anything like rational in the strict sense. that's why game theory is more useful as a normative device to understand system dynamics than a prescriptive one to get guidance on what to do.

"(I think that positing that the Predictor is infallible, and yet doesn't know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can't happen, so you can't ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)"
I don't understand why that's an impossible gedanken. The Predictor can just set up a separate system to do the predictive analysis (another superintelligence, perhaps) and only output the expected decision of the player. I can find a mathematical result using a calculator, and react to the result without knowing the underlying reasons why that result is right.

David Ellis,

Yes, one would think that self-report would make the happiness findings unreliable. But self-report correlates very will with 1) estimates of subject happiness by co-workers and associates, 2) measured serum serotinin levels, and 3) suicide, hospitalization and alcoholism rates. If you want a general introduction to that literature Robert Frank has a very good book on consumption and happiness called, um, Luxury Fever. The first couple chapters are all about the problem of measuring happiness. But self-report matches other measures so closely that it is uncontroversial to use it in the cog psych literature I am familiar with. You could also go to Will WIllkinson's site and search his archives. He blogs a lot about the happiness literature.

And even if you don't believe that, regular church attenders live on average 2-3 years longer. That is public data that can't be faked.

Yes, the sex stuff has a confounding variable of marriage. But if religion leads to marriage and marriage to sex, then the association still stands.

You mention some interesting stuff about inter alia belief and child molesting. I am not familiar with that. I think I would have heard of it. But maybe I need to pay more attention to the literature. Could you please provide some cites to back up your claim? The doctor study you mention would be especially surprising. I would love a cite for that. Paul Bloom is an atheist psychologist at Yale who has done a lot of research on the charity gap (and volunteering, blood donation, etc) between believers and atheists. He has a bunch of episodes on bloggingheads talking about that. If you know of data that disproves his studies you could make quite a name for yourself.

"I don't think there's any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins."

No it doesn't. There are two issues here: the motive for repentance, and the time of repentance. If God rejects anyone who repents solely to avoid going to Hell, then the rationalist who repents out of rational utility maximization and the irrationalist who repents out of raw fear are both damned.

If God accepts repentance performed solely to avoid Hell, but refuses to accept it as part of a plan to wait as long as possible, then a rationalist will take this into account and realize the rational decision is not to make a plan to wait as long as possible. Ze can get that result by simple multiplication and comparing expected utilities.

It's true that a person who irrationally waits as long as possible and then by sheer luck has a change of heart at the last minute will do better than the rationalist, but there's a low chance of having the sincere change of heart at the last minute, and the rationalist knows this. It's like the lottery: rationalists who avoid the lottery will end up worse off than the rare irrationalist who plays and wins the lottery, but should on average expect to end up better off by not playing than by playing. If the rationalist knows that honestly having a sincere change of heart at the last minute is actually very likely, ze may take this into account and decide to wait for it.

So I don't see this scenario as providing an example of irrational agents winning at all. Therefore, I propose a better scenario:

"Consider a universe in which God gives people $50 every time they make an irrational decision. Deliberately making irrational decisions for rational reasons does not count as an irrational decision. Maximize your utility in this universe."

In this scenario, irrational people really do succeed more. But I don't think it proves some potential flaw in rationality. The flaw is considering rationality as a choice, rather than a method of deciding between choices. As if there were two buttons in my brain, one reading "Behave rationally" and the other reading "Behave irrationally", with a little technician inside trying to figure out which button to press, with an infinite regress of buttons inside the technician's head. In any world where God penalized you for pressing the first button, then the best choice really is to press the second button.

But we don't choose that way. The buttons press themselves. In this scenario, my rationality button self-presses and returns the result "The best choice is to have pressed the irrationality button two seconds ago, but it's too late now and it was never your choice anyway." The rationality button has returned the correct result, the winning result, the optimum result, as ideally it always should. Unfortunately, it's useless under the circumstances.

Yvain,

I think your example of the button pusher in your head pressing rational or irrational is really great. But I think we do make that choice.

Examples: When you are mad at how somebody treats you, you can try to calm down or go with it and act nuts. That is the basis of the maximizing strategy in an iterated ultimatum game. You can try to inculcate love for an aging spouse, by going on romantic weekends and whatever, or go looking for someone younger. You can try to foster irrational religious beliefs within yourself. I think it is standard for rabbis to tell people who complain about skepticism, "act like you believe and eventually you will."

I suppose you can posit an infinite regress, but I think in my head there is only one level.

I am not saying it always works. I have tried hard to make myself a sincere believer, with limited success. But I have had a lot of successes with cultivating irrationalities too.

"Rationality" just means the habits of thought that result in win.

if your algorithm's performance is worse than random, inserting a randomly decided variable should improve performance. can this random variable be called rationality?

even if we average the behaviors of all successful people we don't necessarily have enough trials to say with any confidence that these behaviors are rational given their environment.

The talk of the optimal strategy in this scenario doesn't make any sense. Whatever you're measuring, it's supposed to be infinite after death and finite during life. So if your strategy gives you 1,000,000 utilons and then gets you into heaven, and your neighbor's strategy gives him -1,000,000 utilons and then gets him into heaven, then you both tie.

Carl: I don't think the Predictor can be infallible without knowing something about your algorithm. Saying that the Predictor will use a proxy Predictor is equivalent to partitioning the Predictor's mind in two, so I still consider it to be the Predictor knowing your algorithm. But, as Patrick pointed out, that doesn't matter for the purpose that I originally brought it up; my argument was wrong.

Sternhammer: "You also claimed that "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life." False. Try and produce some support for modern churches teaching that. You will fail."

I've been in many churches that taught that. I've been in no churches that didn't teach that. By "modern" churches, you probably mean what I would call "liberal" churches.

As I've already acknowledged twice, there are plenty of churches who teach that once you're saved, you stay saved. And, as I've already pointed out twice, that doesn't matter WRT my post. Let's please not discuss theology anymore on this thread.

David: "When the stakes are either eternal joy or eternal agony, it seems to me that the optimal strategy is to be unselfish all your life."

Yes, but I'm trying to conduct a thought-experiment here.

Yvain wrote: "It's true that a person who irrationally waits as long as possible and then by sheer luck has a change of heart at the last minute will do better than the rationalist, but there's a low chance of having the sincere change of heart at the last minute, and the rationalist knows this."

The person who fears Hell, and discounts the future at a high rate, has a high chance of doing this. There is no tradition saying that God won't accept repentance done out of fear. Convincing people to repent out of fear is an ancient and strong Christian tradition. It is puzzling how to distinguish this from deliberate planning; but the distinction exists. It's the same distinction as between premeditated and 2nd-degree murder. Irrational murder is considered more forgivable than rational murder.

The rest of what Yvain said, I generally agree with. It highlights some confusion in what I originally wrote, and also drags in free will vs. determinism. Can we say that rationality isn't always the best policy, but it's the best policy for choosing a policy? That sounds like what I believe, but I'm not convinced that it makes sense. I don't think I can resolve that question.

What I believe I have resolved is the question as to whether rationality demands that a rational player win the Newcomb game, on the basis that rationality always wins. The "Augustine paradox" is close enough to the Newcomb paradox, that I can say, If the Newcomb paradox is a situation in which rationality must win, then the Augustine paradox is a situation in which rationality must win. And it doesn't. Therefore, it isn't right to assume that rationality must win in the Newcomb paradox.

When discussing whether rationalists should "win" it's important to distinguish them from three other types of "winners":

- Lucky or well-placed fools. Random variation and different starting conditions imply that lots of irrational agents will win for no forseeable reason. If their one success is large enough, or their starting condition good enough, these agents may appear to remain successful even if the decisions they make are no better than chance.

- Unintentionally correct fools. Successful agents being more influential, an unconcious evolutionary process can be observed in memetic development. Irrational agents will collect a semi-random assortment of common biases, heuristics, and emotional triggers, and for many agents these will encourage relatively optimal behavior in many situations, even though the agent has no deliberate reason for possessing these heuristics and could not justify their existence. Many traditional social behaviors fall in this category.

- Intuitively insightful fools. Agents with no conscious attempt at rationality, but possessing an intuitive, meta-level understanding of the behaviors of other agents in their social environment, sufficient to work within the environment to obtain more optimal results.

Irrational agents who do not belong to one of these groups (typically, those in social groups with a higher prevalence of maladaptive heuristic memes) are essentially never "winners".

Note that a "winning" rationalist should, as a first order strategy, seek to emulate the actions (but not the mental processes) of the second type of irrational agent. Higher approximations can deviate from the pseudo-irrational baseline if and only if the rational agent perceives a net gain from the behavior compensating for any social cost resulting from deviating from the behavioral norms.

A rationalist would do well to identify the third type of irrational agent and observe their behavior, as it contains valuable insights on how to improve on the first order stategy above; but this carries the risk of mistaking the first type for the third and being led astray.

soulless automaton: that deserves to be (expanded into) a top level post.

The "Augustine paradox" is close enough to the Newcomb paradox, that I can say, If the Newcomb paradox is a situation in which rationality must win, then the Augustine paradox is a situation in which rationality must win. And it doesn't. Therefore, it isn't right to assume that rationality must win in the Newcomb paradox.

Your "Augustine paradox" is essentially a game in which it is a rule that simply making a plan to win is a losing move. There's nothing like that rule in the Newcomb paradox.

Cyan wrote:

Your "Augustine paradox" is essentially a game in which it is a rule that simply making a plan to win is a losing move. There's nothing like that rule in the Newcomb paradox.
True. Still orthogonal to my point. Perhaps this is a better statement: If one claims that rationality must win in the Newcomb paradox, because rationality always wins in reality; one must also claim that rationality wins in the "Augustine paradox", because the "Augustine paradox" is at least as plausible as the Newcomb paradox. But rationality doesn't win in the Augustine paradox.

I have been rereading Schelling's "Strategy of Conflict" and I still think his claimed benefits for irrational behavior in deterrence situations is not compelling. A better strategy would be a rational imitation of "irrational" (meaning less predictable, and often violent) behavior where it would be of benefit, keeping in mind not just the negative consequences of such behavior in the current situation but also the deterrent effect for future situations. For example, if attacked, retaliate immediately with utmost viciousness, not just for current defense, but more importantly so that no one will attack you in the future (eg, Afghanistan; unfortunately Iraq undercut the deterrent effect of Afghanistan, since it indicated the US might attack you even if you didn't attack the US first). Also, even Schelling put only relatively limited value on apparent irrationality, even for deterrence.

It's not orthogonal -- I should have been less succinct. Here's the slightly longer version: the flaw in the statement "rationality always wins in reality" is that sometimes winning just isn't possible; the correct statement adds the proviso "if possible". Rationality (arguably) doesn't win the Augustine paradox because its rules say no planning -- it directly targets a central feature of rationality. The Newcomb problem does not target rationality, so the outcome of rationality as applied to the Augustine paradox has no relevance to predicting the outcome of rationality as applied to the Newcomb problem.

When I was in diviinity school, I had an old prof (a tanker in WWII, no less) whose ideas of Muslim conversion were based on this line of reasoning.

He said we should hold a sword up to each individual's throat, get them to convert, then chop off their heads.

Of course, as civil libertarians know, and neocons don't, as sadly shown at Bagram, Abu Ghraib and Gitmo, a sword, or far worse, just puts words on someone's lips and not beliefs on their insides.

It seems to me that the basic point here is that as follows.
a) The Newcomb paradox is a paradox because it is a situation where agents are penalized specifically for following a particular strategy mandated by a particular conception of rationality.
b) This is used to argue that the conception of rationality that is being penalized is not "really rational" as a "really rational" strategy would involve responding to the fact that one is in a situation where that behavior is penalized and acting accordingly.
c) But ANY pattern of behavior can be specifically penalized by an outside agent. For instance, a "counterfactual predictor" who put the money in both boxes only if you would two box in Newcomb is not obviously less plausible than a Newcomb predictor.
d) Yet to win in the above situation you should be the sort of person who two-boxes in Newcomb. No behavioral strategy always wins.

Mike - I think that the Newcomb paradox assumes you know ahead of time that you're going to be faced with the Newcomb Predictor, and get to optimize your decision strategy for it. Otherwise it doesn't present a paradox; you just lose.

Cyan wrote:

Here's the slightly longer version: the flaw in the statement "rationality always wins in reality" is that sometimes winning just isn't possible; the correct statement adds the proviso "if possible".
But the irrational, short-sighted, emotional person regularly wins at the Augustine paradox. So winning is possible.
The Newcomb problem does not target rationality, so the outcome of rationality as applied to the Augustine paradox has no relevance to predicting the outcome of rationality as applied to the Newcomb problem.
I'm not trying to predict the outcome of rationality in the Newcomb problem. I'm not claiming that rationality loses in the Newcomb problem. I don't think I can state what I'm trying to show any clearer than I already have.

"I'm not entirely happy with this, because the problem assumes that God cares not just about what you do, but also why you do it, and sends you to Hell if you adopt a strategy for the purpose of winning."

"It explicitly doesn't handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output."

From my guess about Eliezer's theory (http://www.acceleratingfuture.com/tom/?p=159), it should be able to handle God rewarding you for being in a state of mind where you will wind up repenting, or even God rewarding you for belief in Cthulhu (assuming Cthulhu doesn't exist). The problem with this is that it creates an infinite recursion; if you try to use a second-order algorithm to determine which first-order algorithm to use, God will see that you're making second-order decisions based on utility for yourself and send you to Hell. If you use a third-order algorithm to choose a second-order algorithm, God sees it and sends you to Hell, etc., etc.

Tom, your guess is wrong. (I observe that you are usually wrong when you try to guess things that I haven't said; you should be aware of this.) My theory is first-order only - that's kinda the point, that it doesn't need to modify itself because it's just doing the right thing to begin with, that is, it is reflectively stable. It handles Newcomblike problems where other agents care what you will do and can predict what you will do, but the other agent must behave invariantly with respect to your algorithm once this information is screened off; the theory doesn't handle agents who care about arbitrary features of your source code, not without further reflection.

So winning is possible.

But not for a rational agent, by construction. This is similar to the Godel-type sentence, "Cyan cannt prove this sentence." You can prove it easily, but I never can, even though it is a true sentence. (But don't take my word for it...)

Suppose the postulate were that you always win at checkers, because you always win every game it is possible for you to win. Must one therefore also claim that you must win in a game with the rule "People with a 'z' in their last name lose automatically"? Does the fact that you lose at this game have any bearing at all on the postulate?

I've done my best to preserve the phrasing of your comment at 10:17PM in the hopes of making my point congruent with yours.

Phil,

I am not referring to liberal churches. I am referring to the catholic and conservative evangelical churches that I have attended as a member (switched after getting married). Their doctrine explicitly disavows what you attribute to them.

You say that you have been in the churches and that is what they believe. I disagree. I think that is what you thought or heard when you were there, not what they were actually saying. An anecdotal, gestalt sense from an unsympathetic visitor is not the best guide to what doctrine any group actually believes. You are generally right that repentance and forgiveness are big topics. I think you extrapolated to what you thought were the logical conclusions, but they don't think are.

If you want to Overcome Bias, we need to test whether one of us was inaccurate in understanding what was heard. That is why I think we should go to citing written doctrine, which they will helpfully provide on their websites. They also produce enormous amounts of books and articles which you could quote to make your point. You are not making that move. I wonder why.

Billswift: your idea that rational people could get most of the benefits of irrational deterrence sounds great in theory. It breaks down in practice. There were a bunch of interesting studies on lie detection. Most people are very weak in detecting factual lies (something like 52% accuracy when 50-50 is the baseline). But people are awesome at detecting misrepresentation of emotional states (like 90%). Have you ever tried to pretend that you weren't upset and someone knew that you actually were? That's your problem. Evolution has been working on this problem for millions of years, and we have developed a lot of hardware and software for handling it.

That's why FBI and CIA manuals about lie detection focus on emotional states, not factual errors. They look for your move, because it is easier to spot.

It turns out the best way to fake irrational states is to actually have them. And people are great at getting themselves into irrational states.

One example of this is in mating. Females invest much more labor and bodily integrity in reproduction. So they sift the population for males who are irrationally loyal to females and children. So some males try to fake irrational attachment (we call it love). Now maybe you are a great faker, and some guys out there can falsely talk a lot of girls into bed. But the world we observe is one in which detectible irrational attachment is strongly favored, and exists everywhere.

Eliezer wrote:"My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you'll make in the future, such as Newcomb's Problem, Parfit's Hitchhiker and of course the Prisoner's Dilemma. It explicitly doesn't handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output."

This is not the situation we have been in during evolution. Or at least nature cares what algorithms you use when deciding your survival, it doesn't have arbitrary dependencies on the the code as such. E.g. an animal that uses quicksort vs insertion sort for its sorting needs will use less energy thinking and have more available for other purposes.

Phil: your eternal fate depends on your state in the last moment of your life

Ignoring the uninteresting question of whether Christians actually believe this... the winning strategy is to do whatever you can to increase the probability of you having a winning "state" at the final moment.

The optimal self-interested strategy is to act selfishly all your life, and then [sincerely] repent at the final moment. ("sincerely" added)

This is the optimal action. But since winning requires sincerity, it is not the optimal strategy. Planning to be insincere for most of your life is not the best way of achieving the goal of actually being sincere at the end.

The optimal strategy is to do whatever you can to become sincere. (If that isn't allowed either, well, you're screwed - the game has been set up so that anyone in your initial state loses.)

I'm not sure "winning" is a useful word except in sports (and other zero-sum games).
The phrase "best available end-state" would be more accurate for real world situations.

Further, irrational males who feel love will tend to outperform rational ones in securing the reproductive labor of discriminating females.

I disagree with that and I'd say dating is one of the best success cases of rationality - there's nothing natural (=similar to original ancestral environment) about that, the so called "players" have made a small private science of out it, and they're just many orders of magnitude better at it than random guys who do it irrationally on genetic intuition.

If you care about genes and not women - you can cheat even harder and become a sperm donor, it can score you >10 kids easily at tiny fraction of effort.


And even if you don't believe that, regular church attenders live on average 2-3 years longer. That is public data that can't be faked.

Regular church attenders vs Unbelievers?

Selecting out the believers who aren't serious about and committed to their principles while not doing the same in regard to nonbelievers seems a shaky methodology.

You are aware, are you not, for example, that there are atheists who attend churches? Quite a lot of Unitarian Universalists are nontheists.

So wouldn't it make more sense to compare atheists who attend church with theists who attend church and see if its not the believing that's the important factor?


You mention some interesting stuff about inter alia belief and child molesting. I am not familiar with that. I think I would have heard of it. But maybe I need to pay more attention to the literature. Could you please provide some cites to back up your claim?

religiosity and racial prejudice:

http://www.jstor.org/pss/3512331

Religiosity and sexual crime:

http://www.springerlink.com/content/6wnv26q344t52hw7/?p=ed5a99b4443a41a5b36114e7acbfc8b3&pi=5

http://schwitzsplinters.blogspot.com/2007/02/do-atheists-commit-less-vicious-sexual.html

Abortion rates in regions of high religiosity:

http://bhascience.blogspot.com/2008/10/religion-and-abortion-facts.html

That's just a few things that a couple of minutes of Google searching turns up. I'm certainly no expert on the topic and don't claim to be. What I am suggesting, though, is that the evidence does not seem to be as clearly in favor of religion being a positive force in society as you have implied. You seem to be cherrypicking the evidence for things that support your preferred position.

Personally, it doesn't matter to me much either way. I don't form my beliefs based on a desire for beliefs that get me more sex or make me live longer---much as I like both of those things. I'm interested in forming my beliefs by methods with the best chance of generating true beliefs and a low chance of generating false ones.


And even if you don't believe that, regular church attenders live on average 2-3 years longer.

Another point about being careful with correlations. What if this is not because religiosity makes you live longer but simply reflects something like "those who attend church regularly drink alcohol, on average, less than those who don't".

If so, as a nondrinking atheist, I'm likely to have the benefit already without having to put irrational beliefs into my head to get it.

We need to show great care that our preferences aren't influencing the way we judge data like this.

My main point is not that religion doesn't have positive benefits in some ways. Maybe it does. In fact, I think it probably does (though I'm far from convinced there aren't secular alternatives that can achieve equally good results). My point is that I don't think you're being very conscientious in regard to how you're evaluating the evidence. I pointed out some very obvious things you really should have though of yourself but seem to have entirely missed in your haste to find positive social and personal benefits with religiosity

"Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins."

This is the flaw in your argument, I believe. Devaluations are akin to valuations, they are simply part of your overarching utility function and are neither rational nor irrational. Someone who prefers 1 utilicron today to 100 tomorrow isn't irrational. The problem is "utilicrons today" and "utilicrons tomorrow" aren't pure utilicron units, they still have to be processed one final time by a utility function to shed the time component. In this case our agent's utility function may well perform the final computation and conclude "1 utilicron today" = "1 utilicron", "100 utilicron tomorrow" = "0.5 utilicron". That agent would be acting perfectly rationally to live in sin its whole life up until the moment it was about to die.

What your argument has really shown then is that multiple rational agents with different utility functions will ultimately arrive at different net utility, and indeed an agent may perform better than another agent even based on that other agent's utility function. That isn't really surprising considering the setup of this problem essentially becomes "If you have Utility Function 1, you lose. If you have Utility Function 2, you win." In my opinion this has nothing to do with being rational or not.

I don't know if anyone has mentioned it yet, but the example used in the original article fails to take into consideration the full christian doctrines. You assume that salvation is the end all. Simply avoiding hell isn't the end goal. Salvation has two other largely significant purposes:

1) The relationship provided (in the Christian context) with Christ enriches existence. Salvation grants the supplicant sonship, access to God via the mediator Jesus Christ, a host of promises, and the gift of the Holy Spirit.

2) There are rewards promised to the Christian that have nothing to do with simply avoiding hell. A Christian's rank in the world created after Jesus returns, the wealth and richness of his life on the new earth is determinant by how much he suffers for Christ's sake.

The bible teaches that the pleasure derived from a "sinful" action is short lived, that ultimately, the better life is lived when a person avoids such short term enjoyments in lui of richer gains later.

Thus, a rational Christian will deny himself and seek to do good works, not because his salvation depends on it, but because doing such works earns him an eternal reward, such rewards being of far greater value than any temporal gains derived from sinful behavior.

Perhaps this is a better statement: If one claims that rationality must win in the Newcomb paradox, because rationality always wins in reality; one must also claim that rationality wins in the "Augustine paradox", because the "Augustine paradox" is at least as plausible as the Newcomb paradox. But rationality doesn't win in the Augustine paradox.

Well, yes, congratulations. Now substitute 'the Augustine Paradox' for 'game contrived such that rationality loses' and reread the last sentence in the paragraph above....

This reminds me of one of Eliezer's posts on the use of random noise in AI. If the environment is not just unknown but actually out to screw you up, strange things happen. If the environment includes an omnipotent entity who targets rational thought, well, hell, think what you want. It's not difficult to think up a scenario in which rationality is punished. You just say 'in my game, you get punished for rational thought.' But it's not that useful, and certainly not revelatory. If you're hoping to prove that there is no universal strategy, then congratulations, trivial job done. But if you really want to impress me, show me a real-world, god-free situation in which (reciprocal) rationality loses. (And no, 'being happy' doesn't cut it.)

sternhammer's comment above (March 1, 06:14 PM) is excellent, and would probably deserve to be elevated to a topic in its own right.

Rationalists should view irrational allegiences such as religion with admiration, not with contempt. To a rationalist, religion should exhibit a similar beauty as do other evolved organs like, say, the inner ear or the liver. Now, it is possible to admire your inner ear as a marvel of evolution while still using it, but it is more difficult to admire religion as the sleek, mean, optimized tool it is and still be religious. That's because religion and rational thought both impose on your cognition, and you will tend to run into mental resonance catastrophes if you try to be religious and be rational about being religious at the same time. Hence the big divide between rationalists and people of faith, but there are, nevertheless, exceptional people who manage to reconcile the two.

But let us not forget that rationalism itself has huge advantages in the competition between societies. The problem is that these may be stolen by irrational groups. Thus, the nuclear bomb isn't possible without a stable tradition of rationalism, and once you have the bomb, you can safely scoff at the irrational theocracies or warrior societies with all their valour, self-sacrifice and loyalty. Until they steal the design from you, that is.

Jason wrote:

Devaluations are akin to valuations, they are simply part of your overarching utility function and are neither rational nor irrational. Someone who prefers 1 utilicron today to 100 tomorrow isn't irrational.
Correct. I was speaking in shorthand. If you believe that rationality always wins, this should not depend on your valuations. Hence, even if you can win at Augustinian repenting by choosing a particular set of values, it doesn't suffice.

sternhammer wrote

I think that is what you thought or heard when you were there, not what they were actually saying. An anecdotal, gestalt sense from an unsympathetic visitor is not the best guide to what doctrine any group actually believes.
I was a fundamentalist evangelical Christian for about 15 years. I know what I'm talking about, in great depth.

Jason Wall wrote:

Simply avoiding hell isn't the end goal.
Yes. But I'm conducting a thought experiment. I don't mean it as an attack on Christianity. Tho I can see how it could be seen that way.

Ben Jones wrote:

But if you really want to impress me, show me a real-world, god-free situation in which (reciprocal) rationality loses. (And no, 'being happy' doesn't cut it.)
As I've posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life, THEN you should also believe that it will win in the Augustinian paradox, which, unlike the Newcomb problem, is believed by many people to be a real-life problem. If you're asking for a real-world, god-free situation (which may be the right thing to do), then you dismiss both paradoxes.

Cyan wrote:

Suppose the postulate were that you always win at checkers, because you always win every game it is possible for you to win. Must one therefore also claim that you must win in a game with the rule "People with a 'z' in their last name lose automatically"? Does the fact that you lose at this game have any bearing at all on the postulate?
A more precise analogy would be: Suppose E says that a rational agent can win at the Newcomb game, because a rational agent wins all games that might take place in the real-world that it is possible to win. I then provide a different game, and show that it is much more likely to be real-world game than the Newcomb game, and show that a non-rational strategy can win at this game, yet a rational strategy doesn't. I'm playing a bit of a trick, because the game is formulated to penalize rationality; but that doesn't violate the conditions E originally laid down, and doesn't step outside the boundary of "real world games that it is possible to win".

This doesn't show that a rational agent can't win at Newcomb. It doesn't show that either game will actually take place in the real world. It doesn't show that rationality is good or bad, or always wins, or doesn't always win. It JUST SHOWS that THE A PRIORI BELIEF THAT RATIONALITY MUST SOLVE THE NEWCOMB PARADOX IS UNJUSTIFIED.

Has anybody understood that this is what I'm trying to say? Anybody?

Tom, your guess is wrong. (I observe that you are usually wrong when you try to guess things that I haven't said; you should be aware of this.)
I have a wonderful proof that a^n + b^n = c^n has no integral solutions when n > 2. Unfortunately I am too busy to write it down. Until I do, please don't try to guess what it might be.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31