5X5 Episode 53

5X5 Episode 53
Anecdotal Evidence
13th January 2009 Transcript Verified
5X5 52	5X5 54
Skeptical Rogues
S: Steven Novella
R: Rebecca Watson
B: Bob Novella
J: Jay Novella
E: Evan Bernstein
Links
Download Podcast
Show Notes
Forum Topic

Anecdotal Evidence

Voice-over: You're listening to the Skeptics' Guide 5x5, five minutes with five skeptics, with Steve, Jay, Rebecca, Bob and Evan.

J: You may have heard this one from your friend, your neighbor, but I know a guy, who knew another guy who saw something that makes me very scared. Oh, it was on a Friday the 13th and suddenly, all of a sudden a scary ghost that crept up on him and threw hot goo all over my shoes, and then suddenly...

S: This is the SGU 5X5, and tonight we're talking about anecdotes. What are they? What are they good for? Anecdotes are basically stories. They are stories told by a person, or persons, without any documentation, without any careful observation. They are in fact defined as a casual observation that is simply reported. They are a favorite staple of belief in all kinds of unscientific ideas and dubious medical remedies, and other things that do not have solid scientific evidence to support them.

R: Remember that the plural of anecdote is not "evidence". Meaning that just because you have a lot of friends that say that "oh, homeopathy works", it doesn't necessarily mean it works.

B: And one of the reasons for that, is that anecdotal evidence is kind of a catch-all for many logical fallacies, including confirmation bias, incorrect cause and 'post-hoc ergo propter hoc'.

S: As well as memory fallacies. The fact that they're not carefully documented means that we're now relying upon somebody's memory. Which is likely to change over time, and in fact it's likely to change specifically in a way to give the story more meaning, while sacrificing factual accuracy.

B: It would be very common for a friend to tell you an anecdote about somebody that they know, and you have a relationship with that person where you trust them, or you've known them for a while, and it's very easy to believe a story that someone tells you, especially a compelling story, about someone that they know, and it lends credence to whatever it is that they're explaining to you. But unfortunately most of these stories are not accurate or based on anything real.

S: Right, as Michael Shermer is fond of saying, "humans are a story-telling animal, we find stories very compelling, even though we shouldn't".

E: Anecdotes and testimonials are found quite often when people are discussing topics of medicine or medical care, and it's almost a red flag, in a sense. If you're coming across something that you're not quite certain about, and somebody is not presenting you with any real scientific evidence but instead they have a lot of anecdotes and testimonials to offer, your red flag should go off, and you should be very cautious about the claims.

R: Anecdotes can actually be the start of a good scientific process. Anecdotes are how we recognize that maybe there's something going on, so if you get a lot of anecdotes that say homeopathy works, then that means that the next step is testing it scientifically so that you can strip away all of the fallacies, all the problems that Bob alluded to earlier, and then see if there's actually something there. And I find that that's often a good way to deal with people who present anecdotes as evidence of things. Often those are kind of difficult to combat because they say "oh, but this happened to me, you can't prove that wrong, this worked for me". I think a nice way of getting around that is to suggest, well, let's start there and now let's explore it further.

S: Rebecca, you're absolutely right, and I do that as well, I say that an anecdote is a way to generate a hypothesis, but it's not a way to test a hypothesis, because it's such low quality evidence and it is subject to so many biases and fallacies. But sure, people have experiences, and a lot of remedies and scientific discoveries have been made by people just experiencing something. But of course for everyone that pans out, and turns out to be true, there's many many hundreds or whatever, that turn out not to be true, to be misleading. How do we know the difference? Well, by doing the rigorous scientific evidence, controlling for all the things that make anecdotal evidence unreliable and weak.

J: And you know, Steve, even after you've done rigorous research that has a scientific standard applied to it, even then you can't even trust all the data, there's analyzing of that data. You know, taking anecdotal evidence as worthwhile is very dangerous.

S: I also like to quote Barry Beyerstein, who said something that I think is absolutely correct. He said that anecdotal evidence leads one to conclusions that they wish to be true, not to conclusions that actually are true.

S: SGU 5x5 is a companion podcast to the Skeptics' Guide to the Universe, a weekly science podcast brought to you by the New England Skeptical Society in association with skepchick.org. For more information on this and other episodes, visit our website at www.theskepticsguide.org. Music is provided by Jake Wilson.