Overcoming Bias

Cloud Seeding Works

Folks have been seeding clouds to induce rain for over a century, but weather variability has made it hard to collect clear evidence that seeding increases rainfall.  Because of this, many consider cloud-seeding to be a psuedo-science.  But the latest Journal of Applied Meteorology and Climatology presents relatively strong support:

An analysis of cloud seeding activity for the period 1960–2005 over a hydroelectric catchment (target) area located in central Tasmania is presented. The analysis is performed using a double ratio on monthly area averaged rainfall for the months May–October. Results indicate that increases in monthly precipitation are observed within the target area relative to nearby controls during periods of cloud seeding activity. Ten independent tests were performed and all double ratios found are above unity with values that range from 5–14%. Nine out of ten confidence intervals are entirely above unity and overlap in the range of 6–11%. Nine tests obtain levels of significance greater than the 0.05 level. If the Bonferroni adjustment is made to account for multiple comparisons, six tests are found to be significant at the adjusted alpha level. Further field measurements of the cloud microphysics over this region are needed to provide a physical basis for these statistical results.

Absence of evidence is not evidence of absence; sometimes it can just take a long time for clear evidence to accumulate.

The intervention and the checklist: two paradigms for improvement

I'm working on a project involving the evaluation of social service innovations, and the other day one of my colleagues remarked that in many cases, we really know what works, the issue is getting it done. This reminded me of a fascinating article by Atul Gawande on the use of checklists for medical treatments, which in turn made me think about two different paradigms for improving a system, whether it be health, education, services, or whatever. The first paradigm--the one we're taught in statistics classes--is of progress via "interventions" or "treatments." The story is that people come up with ideas (perhaps from fundamental science, as we non-biologists imagine is happening in medical research, or maybe from exploratory analysis of existing data, or maybe just from somebody's brilliant insight), and then these get studied (possibly through randomized clinical trials, but that's not really my point here; my real focus is on the concept of the discrete "intervention"), and then some ideas are revealed to be successful and some are not (with allowances taken for multiple testing or hierarchical structure in the studies), and the successful ideas get dispersed and used widely. There's then a secondary phase in which interventions can get tested and modified in the wild. The second paradigm, alluded to by my colleague above, is that of the checklist. Here the story is that everyone knows what works, but for logistical or other reasons, not all these things always get done. Improvement occurs when people are required (or encouraged or bribed or whatever) to do the 10 or 12 things that, together, are known to improve effectiveness. This "checklist" paradigm seems much different than the "intervention" approach that is standard in statistics and econometrics. The two paradigms are not mutually exclusive. For example, the items on a checklist might have had their effectiveness individually demonstrated via earlier clinical trials--in fact, maybe that's what got them on the checklist in the first place. Conversely, the procedure of "following a checklist" can itself be seen as an intervention and be evaluated as such. And there are other paradigms out there, such as the self-experimentation paradigm (in which the generation and testing of new ideas go together) and the "marketplace of ideas" paradigm (in which more efficient systems are believed to evolve and survive through competitive pressures). I just think it's interesting that the intervention paradigm, which is so central to our thinking in statistics and econometrics (not to mention NIH funding), is not the only way to think about process improvement. A point that is obvious to nonstatisticians, perhaps.

An Especially Elegant Evpsych Experiment

Followup to:  Adaptation-Executers not Fitness-Maximizers, The Evolutionary-Cognitive Boundary

"In a 1989 Canadian study, adults were asked to imagine the death of children of various ages and estimate which deaths would create the greatest sense of loss in a parent. The results, plotted on a graph, show grief growing until just before adolescence and then beginning to drop. When this curve was compared with a curve showing changes in reproductive potential over the life cycle (a pattern calculated from Canadian demographic data), the correlation was fairly strong. But much stronger - nearly perfect, in fact - was the correlation between the grief curves of these modern Canadians and the reproductive-potential curve of a hunter-gatherer people, the !Kung of Africa. In other words, the pattern of changing grief was almost exactly what a Darwinian would predict, given demographic realities in the ancestral environment...  The first correlation was .64, the second an extremely high .92."

(Robert Wright, summarizing:  "Human Grief:  Is Its Intensity Related to the Reproductive Value of the Deceased?"  Crawford, C. B., Salter, B. E., and Lang, K.L.  Ethology and Sociobiology 10:297-307.)

Disclaimer:  I haven't read this paper because it (a) isn't online and (b) is not specifically relevant to my actual real job.  But going on the given description, it seems like a reasonably awesome experiment.  [Gated version here, thanks Benja Fallenstein.  Odd, I thought I searched for that.  Reading now... seems to check out on the basics.  Correlations are as described, N=221.]

The most obvious inelegance of this study, as described, is that it was conducted by asking human adults to imagine parental grief, rather than asking real parents with children of particular ages.  (Presumably that would have cost more / allowed fewer subjects.)  However, my understanding is that the results here squared well with the data from closer studies of parental grief that were looking for other correlations (i.e., a raw correlation between parental grief and child age).

That said, consider some of this experiment's elegant aspects:

• A correlation of .92(!)  This may sound suspiciously high - could evolution really do such exact fine-tuning? - until you realize that this selection pressure was not only great enough to fine-tune parental grief, but, in fact, carve it out of existence from scratch in the first place.
• People who say that evolutionary psychology hasn't made any advance predictions are (ironically) mere victims of "no one knows what science doesn't know" syndrome.  You wouldn't even think of this as an experiment to be performed if not for evolutionary psychology.
• The experiment illustrates as beautifully and as cleanly as any I have ever seen, the distinction between a conscious or subconscious ulterior motive and an executing adaptation with no realtime sensitivity to the original selection pressure that created it.

Our Biggest Surprise

We as a civilization know a lot more today than a random hunter-gatherer from fifty thousand years ago knew.  At a fun dinner with Cosmic Variances's thoughtful Sean Carroll last night, I asked:  What have we learned that is the most surprising?  Sean initially answered "quantum mechanics" but I complained that bundles together too many different things we've learned; I instead want to know what single feature of have we learned would most surprise our distant ancestors? 

Sean then suggested non-determinism, that quantum mechanics appears to suggest that the past does not determine the future.  I suggested what would most surprise our distant ancestors is how big is our universe.  It is big in time and space, in extent and detail, and in the range of things that can fill this extended detailed spacetime. 

So what would you say has been our biggest surprise, weighing not just raw info but also that info's relevance? 

Added:  OK, I see two related surprises, one empirical and one logical. The empirical surprise is that the universe really is big.  The logical surprise is that a big enough universe with a small number of simple essenses can reproduce all of the complex local phenomena that one might otherwise explain via design or a large number of essences.  So per Eliezer and Julian, enough inanimate objects can produce animate object behavior, and per Jed enough incremental adjustments can produce bio and social order.

More added:  Sean remembers the conversation a bit differently; he's probably right. He also asks "the complementary question: what is the most surprising thing about the universe that we haven’t yet discovered, but plausibly could?"

Alien Bad Guy Bias

The Bad Guy Bias applies to Earth signals to aliens.  From the NYT:

The makers of the new movie "The Day the Earth Stood Still" have arranged for it to be beamed into space on ... the same day the movie opens here on planet Earth. ... Dr. Shostak, who was a consultant for the new movie ... [says] there are some people, he acknowledges, who might worry that broadcasting "The Day the Earth Stood Still" could be inimical to our interests. He added, "I think that if these people are truly worried about such things, they might best begin by shutting down the radar at the local airport."

Shostak is right; compared to intentional signals, unintentional signals are a million times larger:

There are three large-dish instruments in the world that are currently employed for doing radar investigations of planets, asteroids and comets: ART (Arecibo Radar Telescope), GSSR (Goldstone Solar System Radar), and EPR (Evpatoria Planetary Radar). Radiating power and directional diagram of these instruments is so outstanding that it also allows us to emit radio messages to outer space, which are practically detectable everywhere in the Milky Way. This dedicated program is called METI (Messaging to Extra-Terrestrial Intelligence) ...

Test Near, Apply Far

Companies often ask me if prediction markets can forecast distant future topics.  I tell them yes, but that is not the place to test any doubts about prediction markets. To vet or validate prediction markets, you want topics where there will be many similar forecasts over a short time, with other mechanisms making forecasts that can be compared. 

If you came up with an account of the cognitive processes that allowed Newton or Einstein to make their great leaps of insight, you would want to look for where that or related accounts applied to more common insight situations.  An account that only applied to a few extreme "geniuses" would be much harder to explore, since we know so little about those few extreme cases.

If you wanted to explain the vast voids we seem to see in the distant universe, and you came up with a theory of a new kind of matter that could fill that void, you would want to ask where nearby one might find or be able to create that new kind of matter.  Only after confronting this matter theory with local data would you have much confidence in applying it to distant voids.

It is easy, way too easy, to generate new mechanisms, accounts, theories, and abstractions.  To see if such things are useful, we need to vet them, and that is easiest "nearby", where we know a lot.  When we want to deal with or understand things "far", where we know little, we have little choice other than to rely on mechanisms, theories, and concepts that have worked well near.  Far is just the wrong place to try new things.

There are a bazillion possible abstractions we could apply to the world.  For each abstraction, the question is not whether one can divide up the world that way, but whether it "carves nature at its joints", giving useful insight not easily gained via other abstractions.  We should be wary of inventing new abstractions just to make sense of things far; we should insist they first show their value nearby. 

Animal experimentation: morally acceptable, or just the way things always have been?

Following the announcement last week that Oxford University’s controversial Biomedical Sciences building is now complete and will be open for business in mid-2009, the ethical issues surrounding the use of animals for scientific experimentation have been revisited in the media—see, for example, here , here, and here.

The number of animals used per year in scientific experiments worldwide has been estimated at 200 million—well in excess of the population of Brazil and over three times that of the United Kingdom. If we take the importance of an ethical issue to depend in part on how many subjects it affects, then, the ethics of animal experimentation at the very least warrants consideration alongside some of the most important issues in this country today, and arguably exceeds them in importance. So, what is being done to address this issue?

In the media, much effort seems to be devoted to discrediting concerns about animal suffering and reassuring people that animals used in science are well cared for, and relatively little effort is spent engaging with the ethical issues. However, it seems likely that no amount of reassurance about primate play areas and germ-controlled environments in Oxford’s new research lab will allay existing concerns about the acceptability of, for example, inducing heart failure in mice or inducing Parkinson’s disease in monkeys—particularly since scientists are not currently required to report exactly how much suffering their experiments cause to animals. Given the suffering involved, are we really sure that experimenting on animals is ethically justifiable?

In attempting to answer this question, it is disturbing to note some inconsistencies in popular views of science. Consider, for example, that by far the most common argument in favour of animal experimentation is that it is an essential part of scientific progress. As Oxford's oft-quoted Professor Alastair Buchan reminds us, ‘You can’t make a head injury in a dish, you can’t create a stroke in a test tube, you can’t create a heart attack on a chip: it just doesn’t work’. Using animals, we are told, is essential if science is to progress. Since many people are apparently convinced by this argument, they must therefore believe that scientific progress is something worthwhile—that, at the very least, its value outweighs the suffering of experimental animals. And yet, at the same time, we are regularly confronted with the conflicting realisation that, far from viewing science as a highly valuable and worthwhile pursuit, the public is often disillusioned and exasperated with science. Recently, for example, people have expressed bafflement that scientists have spent time and money on seemingly trifling projects—such as working out the best way to swat a fly and discovering why knots form—and on telling us things that we already know: that getting rid of credit cards helps us spend less money, and that listening to very loud music can damage hearing. Why, when the public often seems to despair of science, do so many people appear to be convinced that scientific progress is so important that it justifies the suffering of millions of animals?

Behold Our Ancestors

A community of the bacteria Candidatus Desulforudis audaxviator has been discovered 2.8 kilometres beneath the surface of the Earth in fluid-filled cracks of the Mponeng goldmine in South Africa. Its 60C home is completely isolated from the rest of the world, and devoid of light and oxygen. ... 

99.9% of the DNA [there] belonged to one bacterium, a new species. The remaining DNA was contamination from the mine and the laboratory. ...  A community of a single species is almost unheard of in the microbial world. ... Deep-sea vent communities, for instance ... use oxygen ... produced by photosynthesising plankton at the surface. ....

Intelligent Design Honesty

The excellent and famous philosopher Thomas Nagel on teaching intelligent design:

When ... in response to the finding that the teaching of creationism in public schools was unconstitutional, the producers of creation science tried to argue that young earth creationism was consistent with the geological and paleontological evidence, ... their arguments were easily refuted. ... That is a good enough reason not to teach it to schoolchildren. ..

I agree with Philip Kitcher that the response of evolutionists to creation science and intelligent design should not be to rule them out as "not science." He argues that the objection should rather be that they are bad science, or dead science: scientific claims that have been decisively refuted by the evidence. ... However, the claim that ID is bad science or dead science may depend ... on the assumption that divine intervention in the natural order is not a serious possibility. ...

So far as I can see, the only way to make no assumptions of a religious nature would be to admit that the empirical evidence may suggest different conclusions depending on what religious belief one starts with, and that the evidence does not by itself settle which of those beliefs is correct, even though there are other religious beliefs, such as the literal truth of Genesis, that are easily refuted by the evidence. I do not see much hope that such an approach could be adopted, but it would combine intellectual responsibility with respect for the Establishment Clause. ...

'Anyone who thinks the Large Hadron Collider will destroy the world is a t**t.'

This week is Big Bang Week at the BBC, with various programmes devoted to the switch-on of CERN’s Large Hadron Collider (LHC) on Wednesday morning.  Many of these programmes are covered in this week’s issue of the Radio Times—the BBC’s listings magazine—which also features a short interview with Professor Brian Cox, chair of particle physics at the University of Manchester. Asked about concerns that the LHC could destroy the earth, he replies:

‘The nonsense you find on the web about “doomsday scenarios” is conspiracy theory rubbish generated by a small group of nutters, primarily on the other side of the Atlantic.  These people also think that the Theory of Relativity is a Jewish conspiracy and that America didn’t land on the Moon.  Both are more likely, by the way, than the LHC destroying the world.  I’m slightly irritated, because this non-story is symptomatic of a larger mistrust in science, particularly in the US, which includes things like intelligent design. [… A]nyone who thinks the LHC will destroy the world is a t**t.’ (Final word censored by Radio Times.) [1]

Who counts as a nutter and a t**t on this reckoning?  It is true that anyone who thinks there is a 100% chance that the LHC will definitely destroy the world is confused—but it’s probably also true that not many people really think this.  On the other hand, if anyone who thinks that it is worth taking seriously the (admittedly very slim) possibility that the LHC will destroy the world is a t**t, then there are many apparently very clever t**ts knocking about in our universities.  Among these are several of my colleagues: Nick Shackel has previously blogged about the risks of turning on the LHC, as has Toby Ord; and Rafaela Hillerbrand, Toby Ord, and Anders Sandberg recently presented on this topic at the recent Future of Humanity Institute-hosted conference on Global Catastrophic Risks. And, despite having chatted to each of these people about the LHC at some point or another, I’ve never heard any of them express sympathy for the view that the Theory or Relativity is a Jewish conspiracy or that nobody landed on the Moon.  So, are they t**ts or not?