“Ummm, is there another option?”

Hypothetical situation: You receive a call from an alien survey center (and we’re talking about a space alien, not the illegal kind).

An alien interviewer graciously invites you to take a short survey about regional politics on his planet.

Clearly, this is a wrong number since you don’t live on his planet (which is wonderfully ironic since the alien has technology sophisticated enough to translate his language into yours while also placing an intergalactic call, but that’s beside the point).

Even though you know nothing about politics on the alien’s planet, you try not to disappoint people (you don’t go through two divorces without learning something), and agree to take the survey.

The alien asks you the following question:

Which political party do you think will benefit more from Grazz’t M’ynox?

(A) The Zarks for taking a strong stance against Grazz’t M’ynox

(B) The Teligs for trying to keep Grazz’t M’ynox

I don’t know about you, but knowing nothing about Grazz’t M’ynox or the political parties involved, I’d rather go with the party taking the “strong stance” rather than the party simply “trying to keep.”

By “taking a strong stance,” the Zarks have clearly been positioned as a party of strength, principal, and determination.

Conversely, the Teligs are simply “trying to keep” something. Yawn.

So the answer categories have been framed in a way that makes one option more desirable than the other even if you know nothing of the politics and parties involved.

In the latest Bloomberg survey, Americans were asked a similar question (see page 3, half-way down):

Which party will benefit more if the federal government shuts down?

(A) Republicans for taking a strong stance against government spending

(B) Democrats for trying to keep the government open

In the survey, 45% said Republicans, and 34% said Democrats. Given the different framing in the answer categories, is it any surprise that the “taking a strong stance” party beat the “trying to keep” party?

So that question was loaded to favor Republicans. Not that the survey is slanted in only one direction.

For example, take a look at the question right above it, again on page 3. The question involves the potential government shut-down over budget cuts.

The first answer option is an inflexible, line-in-the-sand statement: 

Deep cuts need to be made, even if it means shutting down the government for a while to get that done

The other answer option is a more understanding statement about people working together to find common ground:

Cuts need to be made, but compromises should be reached to avoid a government shutdown

In my experience, the average American is far too busy raising their kids, running their household, and paying the bills to pay close attention to the national economy.

So really, when pressed on such questions, how can busy, hard-working Americans come up with an intelligent answer? Well, how about by listening to the answer options and hedging your bets by picking the one that sounds safest?

And in a situation where you don’t really understand the issue or the implications of your answer, which choice is safer: a hard-line stance or a call for compromise?

So is it really surprising that 77% opted for the “compromise” option while only 20% selected the hard-line stance?

Again, if 1,001 Americans were asked this same question about alien politics, I expect the numbers would work out roughly the same.

I might be deeply ambivalent about Grazz’t M’ynox, but I take a strong stance against questionable survey design, and will not compromise.

On a Scale of 1-to-5, Please Say Something Outlandish

For a nation that doesn’t typically include “survey design” and “statistical analysis” in our education curriculum, we sure are a survey-happy society.

News organizations, in particular, delight in spreading findings on how many people feel this way or that.

But I worry that we’ve become too aware of opinion research and its use in popular culture, and that opinion researchers are exploiting it.

For instance, a new survey from Newsweek finds that 14% of Republican s (+/- 6.8) believe that it’s “Definitely True” that President Obama sympathizes with “Islamic fundamentalists who want to impose Islamic law around the world.”

Another 38% of Republicans say that it is “Probably True.”

Specifically, the question read:

• “Some people have alleged that Barack Obama sympathizes with the goals of Islamic fundamentalists who want to impose Islamic law around the world. From what you know about Obama, what is your opinion of these allegations?”

Clearly, these are highly, highly partisan times, and some Americans disagree vehemently with President Obama’s political philosophy.

But to truly believe he is sympathetic about instituting Islamic Law in the U.S.?

I’m skeptical.

First, I suspect we’re seeing a halo effect in which a person’s overall dislike of something influences every question about that subject. And on highly emotional and controversial topics, this phenomenon is even stronger.

I imagine that the numbers would have been similar if Newsweek had asked Republicans:

• Does President Obama have tentacles?
• Did President Obama turn tail at the Alamo?
• Is President Obama the “man in black” from Lost?

Further, I suspect that people are becoming too media savvy to always be candid on opinion surveys.

With the ubiquity of survey results appearing in the news, I believe that more people understand the connection between survey results and the news cycle, and how surveys can come to frame debates and influence popular opinion.

So I can’t help but wonder how many Republicans heard this question and thought: “Hey, if enough Republicans state belief in this idea, maybe it will get on the news and will help further my personal political goals.”

Twenty years ago, I never would have espoused such a belief.

But in our media savvy culture and highly partisan times, I think it’s a factor at play in political opinion research.

And I also wonder how much opinion researchers know this and exploit it.

So my question for Newsweek is this- did your interviewers follow-up on this question?

Did they probe for the sincerity of this belief and attempt to discover if respondents were expressing general resentment of the president, or even attempting to game the system?

A Rebirth of Push Polling?

Push polling is a scurrilous practice in which partisans posing as opinion researchers ask highly skewed questions, such as, “Can you vote for Candidate XYZ even knowing that he’s committed to enslaving our children?” in order to influence public opinion.

They are easy to identify, and typically have no currency in serious thought.

But I’m worried that there’s a new breed of push polling in which news organizations ask intentionally ambiguous and volatile questions in order to generate sensational headlines.

So what’s the goal of the research…to generate valid opinion data, or to sell more magazines?

I hope the answer is the former, and that we aren’t simply baiting people with strong opinions to make outlandish statements in order to generate a buzz.  

* * * * * * *

Postscript: I’ve noticed that a few on the left have slightly misinterpreted the wording of the question and used some hyperbole in reporting on this survey.

For instance, Eric Kleefeld of TalkingPointsMemo.com described the results this way: “…a majority of Republicans suspect that President Obama wants to impose Islamic law, also known as Sharia, throughout the world…”

Just for clarification, that’s not actually what the question asks, as “sympathizes with” is different from “wants to impose.”

So it’s possible for a respondent to believe that the president sympathizes with fundamentalists, but doesn’t actually want to impose Islamic Law.

I’m nitpicking, but when things are already this sensational, maybe we should nitpick more.

Transparency Matters

The big controversy involving Research 2000 (R2k), a Maryland-based polling company, serves as an excellent reminder that documentation and transparency are crucial to maintaining high credibility.

A prominent R2k client recently fired the company and now alleges that R2k did not provide accurate data. His claim is based on an analysis that suggests a high level of implausibility of R2k’s data, and that implies that R2k fabricated some or all of the data.

R2k vehemently denies the allegations, and they claim that legal ramifications prevent them from fully exonerating themselves at this time.

Transparency & Documentation

We have no idea who’s telling the truth in this one, and The Magnetic Group has no connection to either party.

What we DO know is that there’s a simple remedy that could resolve this issue within the hour: R2k could simply produce the digitally recorded audio conversations of every interview to prove their authenticity.

We digitally record our interviews for every project we undertake. If one of our clients were to request a recording of every interview, we can deliver it within moments.

It’s a simple matter of documentation and transparency.

Sure, it takes time, money, and effort to save and file all of this data. But when you see dust-ups like this one, you realize how important it is.

The Temptation

Thinking beyond this case, it’s clear that temptation exists for research houses to cut their costs by conducting fewer interviewers than they are paid to provide. It’s simple math: if you are paid for 500 interviews, actually performing only 200 interviews will save the research company money.  

And maybe the research company can apply extrapolation and advanced algorithms on the 200 interviews they did conduct to provide good estimates at what the results would have been for a true sample of 500 people.

But unless they are completely transparent about using this technique, it’s fraud. If a research company promises a client a random sample of 500 people, that’s what it needs to deliver.

And research providers need to be ready to offer concrete documentation to prove that they lived up to their obligations.