** **Yesterday I discussed the dilemma of the clever arguer, hired to sell you a box that may or may not contain a diamond. The clever arguer points out to you that the box has a blue stamp, and it is a valid known fact that diamond-containing boxes are more likely than empty boxes to bear a blue stamp. What happens at this point, from a Bayesian perspective? Must you helplessly update your probabilities, as the clever arguer wishes?

If you can look at the box yourself, you can add up all the signs yourself. What if you can't look? What if the only evidence you have is the word of the clever arguer, who is legally constrained to make only true statements, but does not tell you everything he knows? Each statement that he makes is valid evidence - how could you *not* update your probabilities? Has it ceased to be true that, in such-and-such a proportion of Everett branches or Tegmark duplicates in which box B has a blue stamp, box B contains a diamond? According to Jaynes, a Bayesian must always condition on all known evidence, on pain of paradox. But then the clever arguer can make you believe anything he chooses, if there is a sufficient variety of signs to selectively report. That doesn't sound right.

Consider a simpler case, a biased coin, which may be biased to 2/3 heads 1/3 tails, or 1/3 heads 2/3 tails, both cases being equally likely a priori. Each H observed is 1 bit of evidence for an H-biased coin; each T observed is 1 bit of evidence for a T-biased coin. I flip the coin ten times, and then I tell you, "The 4th flip, 6th flip, and 9th flip came up heads." What is your posterior probability that the coin is H-biased?

And the answer is that it could be almost anything, depending on what chain of cause and effect lay behind my utterance of those words - my selection of which flips to report.

- I might be following the algorithm of reporting the result of the 4th, 6th, and 9th flips, regardless of the result of that and all other flips. If you know that I used this algorithm, the posterior odds are 8:1 in favor of an H-biased coin.
- I could be reporting on all flips, and only flips, that came up heads. In this case, you know that all 7 other flips came up tails, and the posterior odds are 1:16 against the coin being H-biased.
- I could have decided in advance to say the result of the 4th, 6th, and 9th flips only if the probability of the coin being H-biased exceeds 98%. And so on.

Or consider the Monty Hall problem:

On a game show, you are given the choice of three doors leading to three rooms. You know that in one room is $100,000, and the other two are empty. The host asks you to pick a door, and you pick door #1. Then the host opens door #2, revealing an empty room. Do you want to switch to door #3, or stick with door #1?

The answer depends on the host's algorithm. If the host always opens a door and always picks a door leading to an empty room, then you should switch to door #3. If the host always opens door #2 regardless of what is behind it, #1 and #3 both have 50% probabilities of containing the money. If the host only opens a door, at all, if you initially pick the door with the money, then you should definitely stick with #1.

You shouldn't just condition on #2 being empty, but this fact plus the fact of the host *choosing* to open door #2. Many people are confused by the standard Monty Hall problem because they update only on #2 being empty, in which case #1 and #3 have equal probabilities of containing the money. This is why Bayesians are commanded to condition on all of their knowledge, on pain of paradox.

When someone says, "The 4th coinflip came up heads", we are not conditioning on the 4th coinflip having come up heads - we are not taking the subset of all possible worlds where the 4th coinflip came up heads - rather we are conditioning on the subset of all possible worlds where a speaker following some particular algorithm *said* "The 4th coinflip came up heads." The spoken sentence is not the fact itself; don't be led astray by the mere meanings of words.

Most legal processes work on the theory that every case has exactly two opposed sides and that it is easier to find two biased humans than one unbiased one. Between the prosecution and the defense, *someone *has a motive to present any given piece of evidence, so the court will see all the evidence; that is the theory. If there are two clever arguers in the box dilemma, it is not quite as good as one curious inquirer, but it is almost as good. But that is with two boxes. Reality often has many-sided problems, and deep problems, and nonobvious answers, which are not readily found by Blues and Greens screaming at each other.

Beware lest you abuse the notion of evidence-filtering as a Fully General Counterargument to exclude all evidence you don't like: "That argument was filtered, therefore I can ignore it." If you're ticked off by a contrary argument, then you are familiar with the case, and care enough to take sides. You probably already know your own side's strongest arguments. You have no reason to infer, from a contrary argument, the existence of new favorable signs and portents which you have not yet seen. So you are left with the uncomfortable facts themselves; a blue stamp on box B is still evidence.

But if you are hearing an argument for the first time, and you are only hearing one side of the argument, then indeed you should beware! In a way, no one can *really* trust the theory of natural selection until after they have listened to creationists for five minutes; and *then* they know it's solid.

We really want to know: what are the typical filters applied in particular areas of life, and thus what evidence does testimony there give us? Doctors, lawyers, parents, lovers, teachers and so on - what filters do they collectively produce on the evidence they get?

Posted by: Robin Hanson | September 29, 2007 at 08:41 PM

We had a related discussion my blog a little while ago - your expert input would be most welcome.

Posted by: Larry Teabag | September 30, 2007 at 05:20 PM

What's being overlooked is that your priors before hearing the clever arguer are not the same as your priors if there were no clever arguer.

Consider the case if the clever arguer presents his case and it is obviously inadequate. Perhaps he refers to none of the usual signs of containing a diamond and the signs he does present seem unusual and inconclusive. (Assume all the usual idealizations, ie no question that he knows the facts and presents them in the best light, his motives are known and absolute, he's not attempting reverse psychology, etc) Wouldn't it seem to you that here is evidence that box B does not contain the diamond as he says? But if no clever arguer were involved, it would be a 50/50 chance.

So the prior that you're updating for each point the clever arguer makes starts out low. It crosses 0.5 at the point where his argument is about as strong as you would expect given a 50/50 chance of A or B.

What lowers it when CA begins speaking? You are predictively compensating for the biased updating you expect to do when you hear a biased but correct argument. (Idealizations are assumed here too. If we let CA begin speaking and then immediately stop him, this shouldn't persuade anybody that the diamond is in box A on the grounds that they're left with the low prior they start with.)

The answer is less clear when CA is not assumed to be clever. When he presents a feeble argument, is it because he can have no good argument, or because he couldn't find it? Ref "What evidence bad arguments".

Posted by: Tom Breton | September 30, 2007 at 05:44 PM

So the prior that you're updating for each point the clever arguer makes starts out low. It crosses 0.5 at the point where his argument is about as strong as you would expect given a 50/50 chance of A or B.I don't believe this is exactly correct. After all, when you're just about to start listening to the clever arguer, do you

really believethat box B is almost certainnotto contain the diamond? Why would you listen to him, then? Rather, when you start out, you have a spectrum of expectations for how long the clever arguer might go on - to the extent you believe box A contains the diamond, you expect box B not to have many positive portents, so you expect the clever arguer to shut up soon; to the extent you believe box B contains the diamond, you expect him to go on for a while.The key event is when the clever arguer

stops talking; until then you have a probability distribution over how long he might go on.The quantity that slowly goes from 0.1 to 0.9 is the estimate you would have

ifthe clever arguer suddenly stopped talking at that moment; it is not your actual probability that box B contains the diamond.Your actual probability starts out at 0.5, rises steadily as the clever arguer talks (starting with his very first point, because that excludes the possibility he has 0 points), and then suddenly drops precipitously as soon as he says "

Therefore..." (because that excludes the possibility he has more points).Posted by: Eliezer Yudkowsky | September 30, 2007 at 05:51 PM

I mostly concur, but I think you can (and commonly do) get some "negative" information before he stops. If CA comes out with a succession of bad arguments, then even before you know "these are all he has" you know "these are the ones he has chosen to present first".

I know that you know this, because you made a very similar point recently about creationists.

(Of course someone *might* choose to present their worst arguments first and delay the decent ones until much later. But people usually don't, which suffices.)

Posted by: g | September 30, 2007 at 07:12 PM

g, agreed.

Posted by: Eliezer Yudkowsky | September 30, 2007 at 07:25 PM

Where do you get that A is "almost certain" from? I just said the prior probability of B was "low". I don't think that's a reasonable restatement of what I said.

It doesn't seem to me that excluding the possibility that he has more points should have that effect.

Consider the case where CA is artificially restricted to raising a given number of points. By common sense, for a generous allotment this is nearly equivalent to the original situation, yet you never learn anything new about how many points he has remaining.

You can argue that CA might still stop early when his argument is feeble, and thus you learn something. However, since you've stipulated that every point raises your probability estimate, he won't stop early. To make an argument without that assumption, we can ask about a situation where he is required to raise exactly N points and assume he can easily raise "filler" points.

ISTM at every juncture in the unrestricted and the generously restricted arguments, your probability estimate should be nearly the same, excepting only that you need compensate slightly less in the restricted case.

Now, there is a certain sense of two ways of saying the same thing, raising the probability per point (presumably cogent) but lowering it as a whole in compensation.

But once you begin hearing CA's argument, you know tautologically that you are hearing his argument, barring unusual circumstances that might still cause it not to be fully presented. I see no reason to delay accounting that information.

Posted by: Tom Breton | September 30, 2007 at 11:13 PM

Tom, if CA's allotment of points is generous enough that the limit makes little difference then it's no longer true that "you never learn anything new about how many points he has remaining" because he'll still stop if he runs out.

If he knows that he's addressing Eliezer and that Eliezer will lower his probability estimate when CA stops, then indeed he'll carry on until reaching the limit (if he can), but in that case what happens is that as he approaches the limit without having made any really strong arguments Eliezer will reason "if the diamond really were in box B then he'd probably be doing better than this" and lower his probability.

Suppose you meet CA, and he says "I think you should think the diamond is in box B, and here's why", and at that instant he's struck by lightning and dies. Ignoring for the sake of argument any belief you might have that liars are more likely to be smitten by the gods, it seems to me that your estimate of the probability that the diamond is in box B should be almost exactly 1/2. (Very slightly higher, perhaps, because you've ruled out the case where there's no evidence for that at all and CA is at least minimally honest.)

Therefore, your suggestion that you lower your probability estimate as soon as you know CA is going to argue his case must be wrong.

What actually happens is: after he's presented evidence A1, A2, ..., Ak, you know not only that A1, ..., Ak are true but also that those are the bits of evidence CA chose to present. And you have some idea of what he'd choose to present if the actually available evidence were of any given strength. If A1, ..., Ak are exactly as good as you'd expect given CA's prowess and perfectly balanced evidence for the diamond's location, then your probability estimate should remain at 1/2. If they're better, it should go up; if they're worse, it should go down.

Note that if you expect a profusion of evidence on each side regardless, k will have to be quite large before good evidence A1 ... Ak increases your estimate much. If that's the case, and if the evidence really does strongly favour box B, then a really clever CA will try to find a way to aggregate the evidence rather than presenting it piecemeal; so in such situations the presentation of piecemeal evidence is itself evidence against CA's claim.

Posted by: g | October 01, 2007 at 05:33 AM

G, you're raising points that I already answered.

Posted by: Tom Breton | October 01, 2007 at 06:15 PM

Only in the sense that you've said things that contradict one another. You said that knowing that you're listening to CA modifies your prior estimate of P(his preferred conclusion) from the outset, and then you said that actually if you stop him speaking immediately then your prior shouldn't be modified. These can't both be right.

I don't see any way to make the "modified prior" approach work that doesn't amount to doing the same calculations you'd do with the "modified estimation of evidence provided by each point made" approach and then hacking the results back into your prior to get the right answer, and I don't see any reason for preferring the latter.

Of course, as a *practical* matter, and given the limitations of our reasoning abilities, prior-tweaking may be a useful heuristic even though it sometimes misleads. But, er, "useful heuristics that sometimes mislead" is a pretty good characterization of what's typically just called "bias" around here :-).

Posted by: g | October 01, 2007 at 07:00 PM