How not to do a survey about evolution.

In my little Darwin Day post last week, I used a chart of acceptance of evolution in various countries.

The data on that chart came from a fairly well-known 2006 study, as reported in Science (pdf). The chart itself has been extensively reproduced since then.

In the comments to that post, commenter "ryan" mentioned seeing something in the Guardian about 50% of Britons rejecting evolution (probably this article). But the chart from the Science article only shows about 20% of Britons rejecting evolution. That's a big change for two years' time!

So what's going on?

The Science study presented a simple statement: "Human beings, as we know them, developed from earlier species of animals", and asked people whether the statement is true or false, or if they didn't know.

They didn't use the word "evolution," which is politically loaded (albeit for idiotic reasons), they just presented a very fundamental aspect of human evolution and asked people what they thought of it.

Asking about human evolution specifically is fine, since people generally have more difficulty accepting that humans evolved from earlier species than accepting that a flatworm did. People who think that the botfly evolved but that humans were made "in God's image" will correctly be included in the anti-evolution or undecided sides.

This is a pretty sensible way to do things. If you want to know if people agree with an idea, ask them about the idea itself. Don't ask them if they agree with the label that the idea has been given, because there's a good chance they don't actually understand what that label means.

Not surprisingly, Science publishes well-done research!

So what about this thing in the Guardian? Well, we're suffering from two problems here. First, the study was badly done. To make matters worse, the reporting on it is misleading (I'll focus on the Guardian piece, but they're far from the only offenders).

The study in question here was part of the "Rescuing Darwin" report. That name should immediately set off alarms. It suggests that they have an agenda beyond just finding out the truth. And while we don't know what that agenda is (yet), such things can skew studies pretty badly.

They can skew studies. But did they?

Hell yes!

In the original study (found here) we have five questions. The first is simply asking what the respondent's age is. The other four are about various theories.

Here are the the theories of evolution religion people were asked about, and the definitions assigned to them by the survey monkeys:

1. Young Earth Creationism is the idea that God created the world sometime in the last 10,000 years.
2. Theistic evolution is the idea that evolution is the means that God used for the creation of all living things on earth.
3. Atheistic evolution is the idea that evolution makes belief in God unnecessary and absurd.
4. Intelligent Design is the idea that evolution alone is not enough to explain the complex structures of some living things, so the intervention of a designer is needed at key stages.

Each of these ridiculously oversimplified and misleading statements was followed by "In your opinion [this theory is] ______" and people were expected to fill in the blank with "definitely true," "probably true," "probably untrue," "definitely untrue," or "don't know."

So, these folks have decided against the sensible method of telling people the idea and having them state whether they agree with it or not in favor of giving definitions for various theories and then asking if people agree with these theories. Oddly, none of these questions even asked people about the actual theory of evolution. They don't even bother to define what the theory of evolution itself is. It's all about religion, and not a bit about science.

The question about "atheistic evolution" is particularly terrible. The other three questions at least refer to real terms, but the only people who use the term "atheistic evolution" are generally wingnuts who then equate it with Stalinism in the next sentence.

Worse yet, if you're an atheist who totally accepts the theory of evolution but doesn't believe it "makes belief in God unnecessary and absurd" (which it arguably doesn't, since it has nothing to say about the supernatural), then you could honestly answer no even to the question about "atheistic evolution"!

Congratulations, you evolution-accepting atheists who think religion and evolutionary theory don't have anything to do with each other, you will now be accused of rejecting evolution thanks to an incredibly poor survey question!

A hint for the survey question-writers: If you're going to do a survey about acceptance of evolution, don't phrase your questions in such a way that someone who accepts evolution gets lumped in with those who don't. Also, maybe ask a question about the actual theory of evolution and not just people's religious beliefs.

It doesn't look like the people behind the survey really cared about getting to the truth, though. They have an agenda, and here it is:

Rescuing Darwin argues that Darwin and his theory have become caught in the crossfire of a philosophical and theological battle in which he himself had little personal interest. On the one side stands a handful of modern Darwinians who insist that evolution has killed God and ideas of design, purpose, morality and humanity. On the other side are their mainly, but not exclusively, religious opponents who, unwilling to adopt such a bleak vision, cite Genesis and Intelligent Design as evidence of evolution's deficiency.

Oh no! A "bleak vision" of life without the Christian God! The book of Genesis and ID as "evidence of evolution's deficiency"! The horrors!

Yeah, this piece of shit was written by Theos, "the public theology think tank." Here's a little about them from "Rescuing Darwin" itself (available as a pdf here).

what Theos stands for
Society is embarking on a process of de-secularisation. Interest in spirituality is increasing across Western culture. Faith is on the agenda of both government and the media. In the arts, humanities and social sciences there are important intellectual developments currently taking place around questions of values and identity. Theos speaks into this new context. Our perspective is that faith is not just important for human flourishing and the renewal of society, but that society can only truly flourish if faith is given the space to do so. We reject notions of a sacred-secular divide.

Well, there's the explanation for why this "study" was about religion instead of science, and also for why it was so poorly done. They don't want people to believe in evolution because they see it as a threat to their little god club. A study showing that people reject evolution supports their agenda, one that shows most people accept it hurts them.

I could spend quite a bit of time tearing apart this shitty "Rescuing Darwin" paper, but it's not worth it. A quick perusal is all you really need to know that this "theological think tank" is exactly what you'd expect from an organization with that title. It's a bunch of morons in fancy robes pontificating on things they know very little about and trying to pass it off as deep thought. Reading it might actually leave you dumber than when you began.

Still, people are allowed to commission surveys if they want. And they can ask really lousy questions to try to lead people to the responses they most desire. Hell, they can even present the "results" of their meaningless survey to the public!

Unfortunately, it takes a gullible and lazy press to move those "results" from their rightful realm of crackpot non-science into the public consciousness. And the Guardian is playing the role of enabler here.

Does the Guardian's story on this all-but-meaningless survey mention the flaws? Does it identify the source as a theological think tank? Does it tell you anything useful about how the study was conducted or what questions were actually asked?

Of course not. Instead it naively reports as fact that:

Half of British adults do not believe in evolution, with at least 22% preferring the theories of creationism or intelligent design to explain how the world came about, according to a survey.

Huh? What about the other 28%? The ones who apparently reject evolution but don't believe in the other bullshit either? Or maybe they don't reject evolution but just got tripped up by the poorly-worded study...

I don't know where that 22% number came from, either. The YEC question got 11% "definitely true" and 21% "probably true", while the ID one got 14% definite and 37% probable. How any of that works out to 22% is beyond me.

I'm going to assume the Guardian's religious affairs correspondent who wrote the story pulled these numbers out of her ass.*

Now, it's entirely possible that a significant percentage of the British people reject evolution. But you can't draw that conclusion from this study, which is, to put it politely, a giant pile of shit.





*Ha ha, the reporter's last name is "Butt"!