We get it all the time, by phone, paper, or online. “Would you like to take a survey?” Then, through the magic of statistics, we learn all sorts of amazing things.
Not so fast.
As Mark Twain (or Disraeli, depending on which Internet quote engine you use) once said, there are three kinds of lies: lies, damn lies, and statistics. And it doesn’t take a lot of learning about surveys to realize how often and how easily statistics can be misused.
Take this week.
The survey itself was actually from January, but I just encountered it a few days ago when it was mentioned in a Facebook feed. “In December, [Idaho Freedom Foundation] commissioned a poll that found that about two-thirds of Idahoans support state-passed legislation that keeps the federal government from dictating whether Idahoans buy health insurance.”
Really? That was interesting. I asked for details. So here’s the question that elicited that response:
Should lawmakers pass legislation that protects Idahoans from being forced to buy health insurance and protects Idahoans from having to enroll in a government insurance program?
Oh, God. Where to start.
1. There is no information about the size of the survey population or where they were drawn from. For statistics to work and be valid, it’s important that the population be of at least a certain minimum size and to be randomly chosen. A survey population where everyone is the same demographic, or from the same area, or who responded to a request for survey participants, isn’t random, any more than asking only Idahoans would give you a clear picture of the entire United States.
2. For a survey to work, the questions need to be objective. If it’s too obvious what sort of answer the surveyor is fishing for, it’s going to skew the results.
So let’s look at the question. It talks about ‘protecting Idahoans’ twice in a single sentence, as well as ‘being forced to buy health insurance’ and ‘having to enroll in a government insurance program.’ Using phrasing like this pretty much guarantees that people are going to say “yes,” because it sounds scary. In fact, if anything surprises me, it’s that the result is only 66 percent.
Around the same time, the IFF did another survey. This one gives a little more information, such as the fact that 400 people were surveyed in six regions of Idaho — meaning they weren’t all picked from a single town, for example. So the population is large enough, and diverse enough geographically, at least, to show some indication that the population may be random.
However, there’s the same problem with the question. Idahoans were asked, The Legislature is getting ready to cut spending this year. Should the Legislature cut taxes as well in an effort to spur economic development, increase personal income and create jobs? Again, about the same two-thirds number agreed with this sentiment.
Well, sure they did. Lots of people are going to agree with a proposal to cut taxes, particularly when they’ve just been told the Legislature is also going to cut spending. Plus, who’s not going to go along with something that’s going to “spur economic development, increase personal income and create jobs?” Hell yes. Sign me up.
I’m not going to debate the issue of whether cutting taxes spurs economic development and creates jobs. The point is, phrasing the question in that way makes it more likely to elicit that response, just like asking “Should the Legislature cut taxes even though it will affect children’s education and make it harder for senior citizens to get medical care?” might elicit a different response.
This doesn’t mean, of course, that Idahoans are in favor of a federal health care plan or are against cutting taxes. What it does mean is that with questions phrased that way, we can’t be sure, and we can’t trust the responses we get to such questions.
Full disclosure: Sharon Fisher is a candidate for the Idaho Legislature, District 21.