When 3 is Greater Than 5

A while back, I was privileged to write a guest post for Vovici’s online research blog after an invitation from industry wizard Jeffrey Henning (he has since joined Affinnova).

While collaborating on a project, Jeffrey and I struck up a conversation on scales, and I mentioned that our company has had tremendous success replacing traditional 5-point scales with 3-point scales.

You can read the original article here, but the main arguments I make in advocating the 3-point scale are:

1. They are visually more appealing to respondents, and thus create more buy-in

2. They generate more variance than 1-5 scales (where ~90% of all respondents say 4 or 5)

3. They decrease response set

All three are important, but from a client’s viewpoint, the main advantage is that a 3-point scale allows us to distinguish between people who are perfectly satisfied with your product/service/brand from people who are REALLY enthusiastic about your product/service/brand.

And that’s crucial.

Interview respondents after they use a 5-point scale and you’ll hear some disconcerting comments that make you really question validity.

People who are extremely enthusiastic about your product/service/brand might only score it as a 4 and argue, “It’s terrific, but things can always be better than they are.”

Other people might assign a rating of 5; and when pressed for their reasoning, they’ll argue, “Well, things are fine; I have no complaints.” Not exactly a ringing endorsement.

In these two examples, you’d like the first person to have assigned a 5, and the second person to have assigned a 3.

I argue that the 3-point scale helps us improve our validity (though certainly not perfect it) by more accurately capturing respondents’ sentiments.

We’ll offer respondents options such as:

• Truly outstanding
• Perfectly acceptable
• Can use some improvement

A few things to note.

First, when using a 5-point scale, respondents with mild quibbles or complaints often feel like a score of 1, 2, or 3 is too harsh. So they will still assign a 4, or a 3 at worst. Thus, with the 3-point scale above, we have made the “negative” category sound less severe to encourage more people to choose it.

Clearly, the people with strong negative opinions (the few who DO use the 1 or 2 on a 5-point scale) will still use our “Can use some improvement” category. But that category now includes many more respondents who aren’t irate or dissatisfied at all, but who do have constructive criticism or minor concerns to share.

Further, we’ve made it more acceptable to answer with the middle category. Whereas respondents using a 1-5 scale feel like a score of 3 is harsh, our middle item is still framed positively.

Thus, we are encouraging more people to use the middle option, which will reduce the number of respondents who use our “best” category even when they really aren’t that enthusiastic.

This should leave us with only our most enthusiastic respondents in the most positive category, all of our “things are fine” respondents in the middle category, and anyone with a complaint in our bottom category, which is exactly what we want.

Is it perfect? Hah. But I think it gets us a little closer to perfection, and that’s a lot in research.

*   *   *   *   *

I recently observed Carbonite using a 3-point scale, which is a perfect choice for them.

They have a pretty simple business model. They back up your data. Either you think they’re doing it or you don’t. No need for 5-point scales here.

So their survey does a few smart things. First, it’s short and clearly formatted, and very unintimidating.

Next, the categories for the closed-ended questions are Satisfied/Neutral/Dissatisfied.

It’s a great choice. Some customers might have actively recovered data using their services and are happy. Others probably have never needed to retrieve files, and they’re neutral, while everyone else probably had an issue and is dissatisfied. Three types of customers, three categories of response.

If you’re a research geek, it’s gratifying to see conceptualization so perfectly match operationalization.



“Ummm, is there another option?”

Hypothetical situation: You receive a call from an alien survey center (and we’re talking about a space alien, not the illegal kind).

An alien interviewer graciously invites you to take a short survey about regional politics on his planet.

Clearly, this is a wrong number since you don’t live on his planet (which is wonderfully ironic since the alien has technology sophisticated enough to translate his language into yours while also placing an intergalactic call, but that’s beside the point).

Even though you know nothing about politics on the alien’s planet, you try not to disappoint people (you don’t go through two divorces without learning something), and agree to take the survey.

The alien asks you the following question:

Which political party do you think will benefit more from Grazz’t M’ynox?

(A) The Zarks for taking a strong stance against Grazz’t M’ynox

(B) The Teligs for trying to keep Grazz’t M’ynox

I don’t know about you, but knowing nothing about Grazz’t M’ynox or the political parties involved, I’d rather go with the party taking the “strong stance” rather than the party simply “trying to keep.”

By “taking a strong stance,” the Zarks have clearly been positioned as a party of strength, principal, and determination.

Conversely, the Teligs are simply “trying to keep” something. Yawn.

So the answer categories have been framed in a way that makes one option more desirable than the other even if you know nothing of the politics and parties involved.

In the latest Bloomberg survey, Americans were asked a similar question (see page 3, half-way down):

Which party will benefit more if the federal government shuts down?

(A) Republicans for taking a strong stance against government spending

(B) Democrats for trying to keep the government open

In the survey, 45% said Republicans, and 34% said Democrats. Given the different framing in the answer categories, is it any surprise that the “taking a strong stance” party beat the “trying to keep” party?

So that question was loaded to favor Republicans. Not that the survey is slanted in only one direction.

For example, take a look at the question right above it, again on page 3. The question involves the potential government shut-down over budget cuts.

The first answer option is an inflexible, line-in-the-sand statement: 

Deep cuts need to be made, even if it means shutting down the government for a while to get that done

The other answer option is a more understanding statement about people working together to find common ground:

Cuts need to be made, but compromises should be reached to avoid a government shutdown

In my experience, the average American is far too busy raising their kids, running their household, and paying the bills to pay close attention to the national economy.

So really, when pressed on such questions, how can busy, hard-working Americans come up with an intelligent answer? Well, how about by listening to the answer options and hedging your bets by picking the one that sounds safest?

And in a situation where you don’t really understand the issue or the implications of your answer, which choice is safer: a hard-line stance or a call for compromise?

So is it really surprising that 77% opted for the “compromise” option while only 20% selected the hard-line stance?

Again, if 1,001 Americans were asked this same question about alien politics, I expect the numbers would work out roughly the same.

I might be deeply ambivalent about Grazz’t M’ynox, but I take a strong stance against questionable survey design, and will not compromise.

Researcher or Parent…You're Still Lame

Parent or researcher…it can be the same thankless job.

Kids want to eat junk food and hardly sleep.

As parents, we know that they need to eat healthy foods and get plenty of sleep so that they can concentrate in school, grow their brains and muscles, and generally achieve a healthier state of being.

And when we do this, our kids say, “Hey, that’s great advice; thanks so much for looking out for me!”

I mean, when we do this, our kids say, “No fair! You’re mean! We will not eat these cold green beans.” [It rhymes if you chant it.]

Do our kids express appreciation and gratitude?

HA!

Again, I say HA!

Sometimes we find ourselves in the same thankless role as researchers, such as when...

• Clients want to use qualitative data to generalize to overall populations.
• Clients want to believe that opt-in surveys are representative of all customers.
• Clients want to think that they can field meaningful 85-question surveys, squeeze the views of 15 people into a 90-minute focus group, and conduct valid research on their own employees.

"Vegetables?!? LAME!"

When clients make questionable decisions, it’s the researcher’s job to look out for their best interests, steer them towards better decisions, and help them understand and embrace alternative options that will ultimately improve their long-term health.

But don’t expect a big hug and an expression of gratitude…at least not immediately.

That only comes later, when the client grows up big and strong, gets interviewed by Oprah, and asserts, “I’d like to thank my researcher for making me eat all those green beans!”  

Hey, I can dream.

"Have you participated in memory loss research in the last 3 months?"

Over on Volition, there's a fun conversation going on about dumb survey questions.

(Volition is an online community for mystery shopping, so many of the posts are from mystery shoppers, panel respondents, and serious survey-takers.) 

I had no idea that so many survey writers add the unforgivable “or not” to the end of questions, as in, “Are you an American citizen or not?”

As one of my favorite bloggers says: “Bad Research! No Biscuit!”

On a Scale of 1-to-5, Please Say Something Outlandish

For a nation that doesn’t typically include “survey design” and “statistical analysis” in our education curriculum, we sure are a survey-happy society.

News organizations, in particular, delight in spreading findings on how many people feel this way or that.

But I worry that we’ve become too aware of opinion research and its use in popular culture, and that opinion researchers are exploiting it.

For instance, a new survey from Newsweek finds that 14% of Republican s (+/- 6.8) believe that it’s “Definitely True” that President Obama sympathizes with “Islamic fundamentalists who want to impose Islamic law around the world.”

Another 38% of Republicans say that it is “Probably True.”

Specifically, the question read:

• “Some people have alleged that Barack Obama sympathizes with the goals of Islamic fundamentalists who want to impose Islamic law around the world. From what you know about Obama, what is your opinion of these allegations?”

Clearly, these are highly, highly partisan times, and some Americans disagree vehemently with President Obama’s political philosophy.

But to truly believe he is sympathetic about instituting Islamic Law in the U.S.?

I’m skeptical.

First, I suspect we’re seeing a halo effect in which a person’s overall dislike of something influences every question about that subject. And on highly emotional and controversial topics, this phenomenon is even stronger.

I imagine that the numbers would have been similar if Newsweek had asked Republicans:

• Does President Obama have tentacles?
• Did President Obama turn tail at the Alamo?
• Is President Obama the “man in black” from Lost?

Further, I suspect that people are becoming too media savvy to always be candid on opinion surveys.

With the ubiquity of survey results appearing in the news, I believe that more people understand the connection between survey results and the news cycle, and how surveys can come to frame debates and influence popular opinion.

So I can’t help but wonder how many Republicans heard this question and thought: “Hey, if enough Republicans state belief in this idea, maybe it will get on the news and will help further my personal political goals.”

Twenty years ago, I never would have espoused such a belief.

But in our media savvy culture and highly partisan times, I think it’s a factor at play in political opinion research.

And I also wonder how much opinion researchers know this and exploit it.

So my question for Newsweek is this- did your interviewers follow-up on this question?

Did they probe for the sincerity of this belief and attempt to discover if respondents were expressing general resentment of the president, or even attempting to game the system?

A Rebirth of Push Polling?

Push polling is a scurrilous practice in which partisans posing as opinion researchers ask highly skewed questions, such as, “Can you vote for Candidate XYZ even knowing that he’s committed to enslaving our children?” in order to influence public opinion.

They are easy to identify, and typically have no currency in serious thought.

But I’m worried that there’s a new breed of push polling in which news organizations ask intentionally ambiguous and volatile questions in order to generate sensational headlines.

So what’s the goal of the research…to generate valid opinion data, or to sell more magazines?

I hope the answer is the former, and that we aren’t simply baiting people with strong opinions to make outlandish statements in order to generate a buzz.  

* * * * * * *

Postscript: I’ve noticed that a few on the left have slightly misinterpreted the wording of the question and used some hyperbole in reporting on this survey.

For instance, Eric Kleefeld of TalkingPointsMemo.com described the results this way: “…a majority of Republicans suspect that President Obama wants to impose Islamic law, also known as Sharia, throughout the world…”

Just for clarification, that’s not actually what the question asks, as “sympathizes with” is different from “wants to impose.”

So it’s possible for a respondent to believe that the president sympathizes with fundamentalists, but doesn’t actually want to impose Islamic Law.

I’m nitpicking, but when things are already this sensational, maybe we should nitpick more.

Did this survey make you think less of us?

Not enough companies understand this simple fact: Surveys are another form of customer communication.

Bad surveys demonstrate poor communication skills and create negative impressions for the parent brand.

So I’m always stunned when I see large, well-known companies field shoddy, poorly crafted surveys that besmirch their image.

Recently, The Weather Channel used an online survey tool to conduct some web research. But survey tools are like any other tool – its success depends a little on the tool, but mostly on the skill of the person using it.

In this case, the person using the tool needed more skill.

One of the early questions on the survey asks the respondent to rate several vacation spots. (See pic below.) However, the final vacation spot listed is, “None of these,” which makes no sense given how this question is formatted.

Okay, just a little oversight. Happens to everyone. But surely the survey won’t make the respondent actually provide a rating for “None of these,” right? I mean, surely someone tested the survey before it hit the field, right?

Wrong. Note the error message below (in red) when I tried to advance without stating my opinion on “None of these.”

And this wasn’t an isolated incident. Other questions demonstrate this same construction error (see pic below).

Finally, the survey also made the unfortunate mistakes of (1) being far too long and (2) punishing respondents for telling the truth.

For every vacation spot in which you expressed any interest, you were “rewarded” by having to fill out this nightmarishly long series of questions (see below).

If you’re like me, you look at all of those options and immediately think:

“No way. I volunteered for this to help you out, I told you that a few of those vacation destinations sounded appealing, and now you’re going to punish me by making me read and answer all of this for every destination?!?!? No way !!!”

I’m sure I wasn’t the only one who just blew through the rest of the survey to get it over with, making the entire experience a waste of everyone’s time and The Weather Channel’s money.

In the end, every person who took this survey ended up thinking that the administrator of the survey was incompetent and unrealistic about their expectations of our time.

And those impressions reflect on The Weather Channel’s overall brand, whether (couldn’t resist) they like it or not.

The Bad Idea Strikes Again!!!

Regular readers are familiar with the bad idea. The bad idea occurs when really smart people decide to tackle a project outside of their area of expertise and eschew the help of experts.

I get the bad idea about home repair projects. “I’ll do it myself and save money!”

But then I end up paying even more money to the plumbers, electricians, and carpenters who need to fix my destruction before they can even mend the original problem.

But I’m in good company.

This morning I was perusing the gamut of news & opinion websites that I read when I came across a reader survey. (Yeah, we’re total suckers for surveys.)

Mind you, this website is run by a bunch of really, really bright folks. Whether you agree with their politics or not, it would be hard to argue that they aren’t a very well informed, brainy bunch.

But their expertise is politics & current events, not survey design.

Example #1

Example #1 is a question about whether or not readers would be willing to pay a fee for advanced content.

 

The answer categories range from “Definitely” to “Definitely Not.” But note the middle option: “I’d need to see more details.”

Two issues here. First, the answer categories are not consistent. Four of them assess direction and degree, while the other (“I need more details”) makes a statement.

 --------------------------------------------------------------

Political Junkie Sidebar

This question is very reminiscent of the misleading exit polls of the 2004 presidential election. The media went wild over the fact that, when asked which single issue mattered most in choosing their presidential candidate, 22% said “Moral Values.”

It edged out the economy/jobs (20%) and terrorism (19%). Suddenly we were inundated with stories about “value voters.”

But the issue wasn’t that clear cut, as the answer categories are simply not consistent with each other. Several are broad issues (e.g., “Health Care” or “Education”), one is a specific issue (“Iraq,”), and one offers an ambiguous concept called “Moral Values.”

Thus, the question inadvertently compares apples to oranges. “Moral values” probably received many votes simply because it is a different animal from all of the other issues-based options.

--------------------------------------------------------------

Back to Example #1. Another issue with this question is that people naturally seek options that they believe will make them most attractive to others (often referred to as “desirability bias").

In this case, there’s obviously one answer category that will make you look smarter than the others. The survey hosts may regret this question when the results come back, as I suspect very few people mark anything other than “I need more details.”

Example #2

In the second example, we have answer categories that are not mutually exclusive. For example, if I have performed any of these actions a single time, there are TWO answer categories that fit my situation: either “once” or “one to three times.”

 

Same thing if I’ve performed one of these actions three times- I have two answer categories that accurately describe my state.  

With so few options, maybe a simple checkbox like this would have been sufficient.  

• Never
• One time
• Two times
• Three or more times

Have any examples of the bad idea that can be shared in a public forum? We'd love to hear 'em!