The Bad Idea Strikes Again!!!

Regular readers are familiar with the bad idea. The bad idea occurs when really smart people decide to tackle a project outside of their area of expertise and eschew the help of experts.

I get the bad idea about home repair projects. “I’ll do it myself and save money!”

But then I end up paying even more money to the plumbers, electricians, and carpenters who need to fix my destruction before they can even mend the original problem.

But I’m in good company.

This morning I was perusing the gamut of news & opinion websites that I read when I came across a reader survey. (Yeah, we’re total suckers for surveys.)

Mind you, this website is run by a bunch of really, really bright folks. Whether you agree with their politics or not, it would be hard to argue that they aren’t a very well informed, brainy bunch.

But their expertise is politics & current events, not survey design.

Example #1

Example #1 is a question about whether or not readers would be willing to pay a fee for advanced content.

 

The answer categories range from “Definitely” to “Definitely Not.” But note the middle option: “I’d need to see more details.”

Two issues here. First, the answer categories are not consistent. Four of them assess direction and degree, while the other (“I need more details”) makes a statement.

 --------------------------------------------------------------

Political Junkie Sidebar

This question is very reminiscent of the misleading exit polls of the 2004 presidential election. The media went wild over the fact that, when asked which single issue mattered most in choosing their presidential candidate, 22% said “Moral Values.”

It edged out the economy/jobs (20%) and terrorism (19%). Suddenly we were inundated with stories about “value voters.”

But the issue wasn’t that clear cut, as the answer categories are simply not consistent with each other. Several are broad issues (e.g., “Health Care” or “Education”), one is a specific issue (“Iraq,”), and one offers an ambiguous concept called “Moral Values.”

Thus, the question inadvertently compares apples to oranges. “Moral values” probably received many votes simply because it is a different animal from all of the other issues-based options.

--------------------------------------------------------------

Back to Example #1. Another issue with this question is that people naturally seek options that they believe will make them most attractive to others (often referred to as “desirability bias").

In this case, there’s obviously one answer category that will make you look smarter than the others. The survey hosts may regret this question when the results come back, as I suspect very few people mark anything other than “I need more details.”

Example #2

In the second example, we have answer categories that are not mutually exclusive. For example, if I have performed any of these actions a single time, there are TWO answer categories that fit my situation: either “once” or “one to three times.”

 

Same thing if I’ve performed one of these actions three times- I have two answer categories that accurately describe my state.  

With so few options, maybe a simple checkbox like this would have been sufficient.  

• Never
• One time
• Two times
• Three or more times

Have any examples of the bad idea that can be shared in a public forum? We'd love to hear 'em!