5X5 Episode 112

This episode needs:  proof-reading,

How to Contribute

5X5 Episode 112
Anecdotal Evidence
2 May 2012
5X5 111	5X5 113
Skeptical Rogues
S: Steven Novella
R: Rebecca Watson
B: Bob Novella
J: Jay Novella
E: Evan Bernstein
Links
Download Podcast
Show Notes
Forum Topic

Anecdotal Evidence

Voice-over: You're listening to the Skeptics' Guide 5x5, five minutes with five skeptics, with Steve, Jay, Rebecca, Bob and Evan.

S: This is the SGU 5x5 and tonight we're talking about anecdotal evidence. Anecdotal evidence is a casual observation – one that is not done under any strict scientific protocol. There are extreme limitations to using anecdotal evidence as evidence, and it does not qualify as scientific evidence for these reasons. For example, anecdotes can not be statistically analyzed, because they are not gathered in a systematic or a thorough way. Therefore they're subject to so-called cherry picking. People may remember or just point to those anecdotes that tend to support something that they wish to believe in. This leads to what we call confirmation bias, selecting only confirming evidence, and dismissing or forgetting disconfirming evidence. There are lots of logical fallacies inherent in anecdotal evidence as well, such as the hasty generalization. Because anecdotes are not controlled or systematic, there is no way to know if anecdotal experiences are typical. Therefore, they– trying to generalize from an anecdote is a form of hasty generalization. So, anecdotal experience is not the same as scientific evidence because a scientific study endeavours to capture all of the information on a relevant question systematically, so that it is possible to do statistical analysis, outcomes are randomized and efforts are made to make sure that there is a representative sample, so that the outcome can be generalized to whatever the population in question is. In short: scientific studies allow for the control of variables, controlling for possible confounding factors, whereas anecdotal experience does not. It's quirky, it's biased, and it is not therefore a source of reliable evidence.

R: That said, there are instances in which a particular type of anecdotal evidence can be useful to science. For instance what we call case studies. Case studies are in-depth analyses of specific events, taking into account the context in which those events happen. Now, this is not in any way to be considered, for instance, a scientific study on its own. However it can be useful in, for instance, falsifying certain hypotheses. Karl Popper, a famous philosopher of science, came up with a famous example in which he said "all swans are white" and then proposed that one single observation of a black swan would be enough to falsify that proposition, and that observation of the black swan would be the case study. A case study is very good for finding those black swans and detailing them, and that type of case study can be very helpful in forming a hypothesis, from which you can then go and do more scientific research using more evidence that you can gather.

S: Yeah, I think a good way to summarize the role of anecdotes in science is that they are useful for generating hypotheses, they are just not useful for testing hypotheses, for that you need scientific data.

E: Steve, another problem with anecdotal evidence has to do with the fallibility of human memory. People are innately poor historians; our memories have limitations and over time our memories tend to decay, or warp, or merge with other memories. In the case, for example, of a person suffering from some sort of medical illness, anecdotes can become contaminated with false memories and exaggerations due to the sensitive and deeply personal nature of the experience. There's a tendency for details to evolve over time to make a story sound more clean and profound. So in a patient's own memory, they might exaggerate certain things, such as the severity of the symptoms prior to the treatment or exaggerate the response to the actual treatment. They might clean up the time-line of events so that improvement began very soon after a certain treatment took place rather than before, or long after. They might forget other treatments that were taken, or distort what they were told by their various health-care providers and so on, and so on. When we're recounting stories about our own suffering of illness, and the remedies involved in our recovery, then to take our anecdotes as some sort of unbiased data is actually little more than a hopelessly flawed body of evidence, both logically and scientifically.

S: Yeah, I think Rebecca was talking about case reports, and they are anecdotal, but they're one notch above the anecdote, in that at least the details are written down and therefore fixed. They are no longer subject to all of the vagueries of human memory that you were talking about Evan.

B: It seems clear that anecdotal evidence is a powerful but flawed driver for belief in people, and as such, has to be strictly controlled in courts of law, as well as normal day-to-day or scientific activities. If someone gives testimony in court that cannot be verified due to its very nature, for example, if somebody says: "The deceased told me that the defendant threatened to kill him", then it's legally called anecdotal, or more often, 'hearsay' evidence. According to the website Laws.com, hearsay is "any information is any information gathered by one party or person from another, concerning a particular event, condition or thing that was not directly related to the accused person". Now how many times on legal shows, guys, have you heard someone say "Your honor, that's hearsay"? It's very common, you hear it all the time. Like science itself, and skepticism as well, the legal system – of the United States, anyway – has a very poor opinion of anecdotal, or hearsay, evidence. So poor, infact, that as a general rule, this type of evidence is not allowed in a civil or criminal trial, as described by the hearsay rule. There are only limited situations when it may be allowed, including when a case makes a very severe charge. Iπn these situations it's usually used to bolster a defense, or perhaps to hasten a verdict. One problem, though, that exists with anecdotal evidence in courts however, is that it can be very fuzzy to define. Sometimes the line between fact and anecdote can be a bit blurred, so it's often up to the courts, or the judge, to determine when to allow some types of this kind of evidence.

J: So as you can see, anecdotal evidence is actually very powerful. As an example, you could do hours of research about something on the internet, say, and then a single comment by a friend or family-member that contradicts the information that you've been reading about could wipe it away. And I happen to have an example that happened to me recently, that illustrates this perfectly, and don't think for a second that I missed the irony in the fact that I'm giving you an anecdote about anecdotal evidence. But I think you'll get the whole thing when I tell you this quick story. I was at work, and I noticed a co-worker had a calorie-burning monitor – like a watch type of thing, and he explained that it tracks all these details about what you do during the day, and it estimates how many calories you burned. So he said he loves it, and he went on saying that it actually helped him lose weight, and we talked about it for about, you know, 5-10 minutes, I was asking him questions. And then afterwards, I was sat there wondering how accurate could it be? And if I should get one. So, about 10 minutes later, another co-worker who works right near there came over, and he said he heard the entire conversation, and that the other guy didn't do a good job at tracking the burned calories. He said that with authority, and in that instant, I convinced myself that he was right, and that I wasn't gonna buy it. His story matched what I suspected, and in that moment, as soon as I heard something that matched what my internal dialogue was, I went with that piece of information, regardless of what the ultimate reality was.

S: So I'd just like to finish with a quote from a friend of mine, who unfortunately is no longer with us – Barry Beyerstein, who said: "Anecdotal evidence leads us to conclusions that we wish to be true. Not conclusions that actually are true".

S: SGU 5x5 is a companion podcast to the Skeptics' Guide to the Universe, a weekly science podcast brought to you by the New England Skeptical Society in association with skepchick.org. For more information on this and other episodes, visit our website at www.theskepticsguide.org. Music is provided by Jake Wilson.

References