« Bay Area Bayesians Unite! | Main | College Choice Futures »

October 28, 2007

Comments

Eliezer, are you familiar with Russell and Wefald's book "Do the Right Thing"?

It's fairly old (1991), but it's a good example of how people in AI view limited rationality.

Maybe you could *exploit* this, if the question you're gathering evidence for is important enough to warrant all that costly searching. Spending hours digging through obscure journals is not something most people do for fun, but if you can come up with a pet theory which needs reinforcing, most people would rather do the evidence-gathering than be forced to give it up.

'Motivated stopping'? What springs to my mind is psi tests.
If you regard psi tests as a possibly infinite series, then when you cut off testing and start analysing can produce any result you want.

'Lucky streaks' can occur at any any time in a string of random numbers.

That's why in psi testing you must calculate the exact number of tests required to show an effect of the size you expect and do precisely that number of tests, no more and no less. And you are not allowed to throw away the tests that resulted in average or negative results either.

My favourite example of motivated stopping is Lazzarini's experimental "verification" of the Buffon needle formula.

(Drop toothpicks at random on a plane ruled with evenly spaced parallel lines. The average number of line-crossings per toothpick is related to pi. Lazzarini did the experiment and got pi to 6 decimal places. It seems clear that he did this by doing trials in batches whose size made it likely that he'd get an estimate equivalent to pi = 355/113, which happens to be very close, and then did one batch at a time until he happened to hit it on the nose.

Completely off-topic, here's a beautiful derivation of the formula: Expectations are additive, so the expected number of line-crossings is proportional to the length of the toothpick and doesn't depend on what shape it actually is. So consider a circular "toothpick" whose diameter equals the spacing between the lines. No matter how you drop this, you get 2 crossings. Therefore the constant of proportionality is 2/pi. Therefore the expected number of crossings for *any* toothpick of length L, in units where the line-spacing is 1, is 2L/pi. If L<1 then this is also the probability of getting a crossing at all, since you can't get more than one.)

To put it differently, motivated stopping is a problem in pi tests just like it is in psi tests. :-)

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31