Wednesday, April 23, 2014
What's New in the New West
Home » Rockies » Idaho » Boise » Another Episode of Damn Lies and Statistics
We get it all the time, by phone, paper, or online. "Would you like to take a survey?" Then, through the magic of statistics, we learn all sorts of amazing things. Not so fast. As Mark Twain (or Disraeli, depending on which Internet quote engine you use) once said, there are three kinds of lies: lies, damn lies, and statistics. And it doesn't take a lot of learning about surveys to realize how often and how easily statistics can be misused.

Another Episode of Damn Lies and Statistics

We get it all the time, by phone, paper, or online. “Would you like to take a survey?” Then, through the magic of statistics, we learn all sorts of amazing things.

Not so fast.

As Mark Twain (or Disraeli, depending on which Internet quote engine you use) once said, there are three kinds of lies: lies, damn lies, and statistics. And it doesn’t take a lot of learning about surveys to realize how often and how easily statistics can be misused.

Take this week.

The survey itself was actually from January, but I just encountered it a few days ago when it was mentioned in a Facebook feed. “In December, [Idaho Freedom Foundation] commissioned a poll that found that about two-thirds of Idahoans support state-passed legislation that keeps the federal government from dictating whether Idahoans buy health insurance.”

Really? That was interesting. I asked for details. So here’s the question that elicited that response:

Should lawmakers pass legislation that protects Idahoans from being forced to buy health insurance and protects Idahoans from having to enroll in a government insurance program?

Oh, God. Where to start.

1. There is no information about the size of the survey population or where they were drawn from. For statistics to work and be valid, it’s important that the population be of at least a certain minimum size and to be randomly chosen. A survey population where everyone is the same demographic, or from the same area, or who responded to a request for survey participants, isn’t random, any more than asking only Idahoans would give you a clear picture of the entire United States.

2. For a survey to work, the questions need to be objective. If it’s too obvious what sort of answer the surveyor is fishing for, it’s going to skew the results.

So let’s look at the question. It talks about ‘protecting Idahoans’ twice in a single sentence, as well as ‘being forced to buy health insurance’ and ‘having to enroll in a government insurance program.’ Using phrasing like this pretty much guarantees that people are going to say “yes,” because it sounds scary. In fact, if anything surprises me, it’s that the result is only 66 percent.

Around the same time, the IFF did another survey. This one gives a little more information, such as the fact that 400 people were surveyed in six regions of Idaho — meaning they weren’t all picked from a single town, for example. So the population is large enough, and diverse enough geographically, at least, to show some indication that the population may be random.

However, there’s the same problem with the question. Idahoans were asked, The Legislature is getting ready to cut spending this year. Should the Legislature cut taxes as well in an effort to spur economic development, increase personal income and create jobs? Again, about the same two-thirds number agreed with this sentiment.

Well, sure they did. Lots of people are going to agree with a proposal to cut taxes, particularly when they’ve just been told the Legislature is also going to cut spending. Plus, who’s not going to go along with something that’s going to “spur economic development, increase personal income and create jobs?” Hell yes. Sign me up.

I’m not going to debate the issue of whether cutting taxes spurs economic development and creates jobs. The point is, phrasing the question in that way makes it more likely to elicit that response, just like asking “Should the Legislature cut taxes even though it will affect children’s education and make it harder for senior citizens to get medical care?” might elicit a different response.

This doesn’t mean, of course, that Idahoans are in favor of a federal health care plan or are against cutting taxes. What it does mean is that with questions phrased that way, we can’t be sure, and we can’t trust the responses we get to such questions.

Full disclosure: Sharon Fisher is a candidate for the Idaho Legislature, District 21.

About Sharon Fisher

Comments

  1. JAYoung says:

    Real opinion research is a science, and valid data can only be obtained by using carefully constructed questions that remove any hint of bias. Respondents must be carefully chosen to reflect the larger whole.
    Push polls on the other hand use biased questions to change the opinions of people as they are being asked the questions. Push polls are a propaganda technique.
    The public rarely draws the distinction: a poll is a poll.
    The news media could help, but most news organizations also use unscientific “polls” to gather garbage data about “public opinion” on their Web pages. A proper survey costs thousands of dollars but cash-strapped newspapers can put up a question on their Web site at no cost and make it look like they’re doing their job.

  2. Tom von Alten says:

    More to the point, statistics don’t lie, but liars use statistics. There are many questions that can’t be answered directly, but the power of statistics enable us to make good inferences. As noted, that takes due diligence.

    The Idaho Freedom Foundation is not a reliable source from everything I’ve seen. When it comes to rooting around and re-publishing public records, I can give them the benefit of the doubt, even as I presume WHICH things the dig up are designed to push their agenda.

    For surveys and stats and the like, I have no reason to believe Wayne and whoever works for him are competent, and I have every reason to believe they’re strongly biased by their ideology. Either fault by itself will produce untrustworthy results; together, they make trustworthy results exceedingly hard to come by.

    But IFF is about lobbying, first and foremost, entertainment to the extent that it serves its first purpose, and promulgating objective information not at all.

  3. Sharon Fisher says:

    Thanks, JAYoung, for using the term of art “push poll”; I’d meant to include that and forgot.

  4. J says:

    Oh my god the way you phrase a question can have a statistically significant impact on poling data!!!!!! What a revelation, I’m sure the Republican Party in a true act of deviance just discovered this. They must be stopped before all statistics become meaningless. Hurry, recent polling data suggests we are running out of time!

  5. Jon Cheever says:

    In a recent poll I took of four women at Starbucks, 75% of the respondents said that they would rather sit on a beach in Fiji than read about statistical validity on NewWest.net. I therefore conclude that most women in America find this article irrelevant.

    -Jon Cheever.

  6. Sharon Fisher says:

    J, I’m not expecting to change the minds of anyone who uses this technique. All I want to do is help people who don’t know about it to recognize when it’s being used.

  7. J says:

    I’m sorry Sharon; I was in a negative mood yesterday, my comment reflects that. Your article was basically about survey design, and your thoughts about a particularly poorly phrased question. I agree the question is terrible, but designing questions to generate a representative data sample can be troublesome. After reading your article I was left wondering, how would have Sharon asked the question?

    “If it’s too obvious what sort of answer the surveyor is fishing for, it’s going to skew the results.”

    This has nothing to do with the problem, and when I read the question it doesn’t become obvious to me what the desired result is…..However. Most Nouns, Verbs, and Adjectives have either positive or negative connotations. For example Child like and childish, they both mean the same thing but one is an insult and one can be a compliment. And as you hinted at the wording is biased although you decide to call it “scary”. A few more problems with the question; depending on what side of the aisle you read it from it can mean two very different things. This is no good, every respondent needs to understand the question the same way. The question is double barreled meaning it asks two separate questions. Am I supposed to answer whether or not I want “lawmakers pass legislation that protects Idahoans from being forced to buy health insurance”, or if I want protection “from having to enroll in a government insurance program?” This is no good. And possibly the biggest problem is the fact that there are only two answers to chose from YES or NO. So here is what I would have done.

    The federal government recently passed legislation requiring everyone enroll in a health insurance plan. How would you feel about the Idaho sate legislature passing a law in order to void this provision?

    Strongly Agree
    Agree
    Don’t know
    Disagree
    Strongly Disagree

    And then if you want you could ask another question about the government insurance program. I find it somewhat interesting that you take them to task over the way the posed the question but didn’t offer a better version. Not to sound like a mean spirited person but next time do more research. There is tons of info out there on best practices for survey design. May your political career take you as far as you can take it.

  8. Sharon Fisher says:

    I have to research it more because I’m not even sure to the extent that it requires people to buy insurance. However, if it does, then I’d ask the question more like the following:

    “Congress recently passed a law intended to provide health insurance to more Americans. Part of the way this is implemented requires Americans to buy health insurance (with financial assistance provided to low-income people). Should lawmakers pass a law stating that Idahoans do not have to follow this requirement?”

    I’m sure it’s not perfect; it’s just what I could come up with at the spur of the moment.

  9. Mickey Garcia says:

    Polls are just another weapon in the political propaganda game. The presidents poll are up. The presidents polls are down. The air head media breathlessly reports polls constantly as if the polls are actually meaningful, based on what? Its like asking millions of people how they’re feeling today. Walk up to someone and ask a public policy question that requires some thought and analysis and attention to critical detail, and chances are the person being questioned will answer off of the top of his or her head based on rumors going around and that person’s prejudices.

  10. Dave Skinner says:

    Push polling is abused by both sides, Sharon.
    Bloomberg’s NRA-owner poll by Luntz was a recent classic, as was Trout Unlimited’s poll about Tester’s wilderness bill. If S-1470 actually did the things the poll question claimed, I’d support it.
    I have to wonder about telephone polls any more….who the heck is around to answer the phone, and d’ya think anyone with a BRAIN is going to sit through a push-poll when it becomes clear?
    Not me.
    I think maybe I’ve been called twice. The first thing I do is ask the questioner to spill for whom the poll tolls. Both times, the poll terminated….

  11. JAYoung says:

    I’ve refused to cooperate with any phone polling for years as a matter of personal privacy — even when it’s a issue that’s important to me.
    Eventually your phone number disappears from the data bases many of the polling companies sell back and forth to each other, and you’re not bothered any more.

  12. Sharon Fisher says:

    JADP, but here’s how a professional pollster asked a similar question:

    Q: Some people believe a federal law requiring every American to buy or obtain health insurance is unconstitutional. If that law passes, some states have announced that they will sue the federal government to fight that requirement. If the health care reform passes, would you favor having your state sue the federal government to prevent it from becoming law?