Five Reasons to NEVER Design a Survey without a Comment Field

The good news is that most researchers do, in fact, understand and appreciate the value of comment data from open-ended questions.

Indeed, many say feedback in consumers’ own words is indispensable.

Among researchers we recently polled:

The good news is that most researchers do, in fact, understand and appreciate the value of comment data from open-ended questions.

Indeed, many say feedback in consumers’ own words is indispensable.

Among researchers we recently polled:

• 70% would NEVER launch tracker OR even an ad-hoc (66%) survey without a comment field
• 80% DO NOT agree that analyzing only a subset of the comment data is sufficient
• 59% say comment data is AT LEAST as important as the numeric ratings data (and many state they are the most important data points)
• 58% ALWAYS allocate time to analyze comment data after fielding

In Their Own Words: “Essential”

In contrast to the flippancy we saw in comments from those who don’t see any need for open-ended survey questions, researchers who value open-ends felt pretty strongly about them.

Consider these two verbatim responses, which encapsulate the general sentiment expressed by researchers in our survey:

“Absolutely ESSENTIAL. Without [customer comments] you can easily draw the wrong conclusion from the overall survey.”

“Open-ended questions are essential. There is no easy shortcut to getting at the nuanced answers and ‘ah-ha!’ findings present in written text.”

As it happens, respondents to our survey provided plenty of detailed and thoughtful responses to our open-ended questions.

We, of course, ran these responses through OdinText and our analysis identified five common reasons for researchers’ belief that comment data from open-ended questions is critically important.

So here’s why, ranked chronologically in ascending order by preponderance of mentions and in their own words…

 Top Five Reasons to Always Include an Open-End

#5 Proxy for Quality & Fraud

“They are essential in sussing out fraud—in quality control.”

“For data quality to determine satisficing and fraudulent behavior”

“…to verify a reasonable level of engagement in the survey…”

#4 Understand the ‘Why’ Behind the Numbers

“Very beneficial when trying to identify cause and effect”

“Open ends are key to understand the meaning of all the other answers. They provide context, motivations, details. Market Research cannot survive without open ends”

Extremely useful to understand what is truly driving decisions. In closed-end questions people tend to agree with statements that seem a reasonable, logical answer, even if they have not considered them before at all

“It’s so critical for me to understand WHY people choose the hard codes, or why they behave the way the big data says they behave. Inferences from quant data only get you so far – you need to hear it from the horse’s mouth…AT SCALE!”

“OEs are windows into the consumer thought process, and I find them invaluable in providing meaning when interpreting the closed-ended responses.”

#3 Freedom from Quant Limitations

“They allow respondents more freedom to answer a question how they want to—not limited to a list that might or might not be relevant.”

“Extremely important to gather data the respondent wants to convey but cannot in the limited context of closed ends.”

“Open-enders allow the respondent to give a full explanation without being constrained by pre-defined and pre-conceived codes and structures. With the use of modern text analytics tools these comments can be analyzed and classified with ease and greater accuracy as compared to previous manual processes.”

“…fixed answer options might be too narrow.  Product registration, satisfaction surveys and early product concept testing are the best candidates…”

“allowing participants to comment on what’s important to them”

#2 Avoiding Wrong Conclusions

“We code every single response, even on trackers [longitudinal data] where we have thousands of responses across 5 open-end questions… you can draw the wrong conclusion without open-ends. I’ve got lots of examples!”

“Essential – mitigate risk of (1) respondents misunderstanding questions and (2) analysts jumping to wrong conclusions and (3) allowing for learnings not included in closed-ended answer categories”

“Open ended if done correctly almost always generate more right results than closed ended.  Checking a box is cheap, but communicating an original thought is more valuable.”

#1 Unearthing Unknowns – What We Didn’t Know We Didn’t Know

“They can give rich, in-depth insights or raise awareness of unknown insights or concerns.”

“This info can prove valuable to the research in unexpected ways.”

“They are critical to capture the voice of the customer and provide a huge amount of insight that would otherwise be missed.”

“Extremely useful.  I design them to try and get to the unexpected reasons behind the closed-end data.”

“To capture thoughts and ideas, in their own words, the research may have missed.”

“It can give good complementary information. It can also give information about something the researcher missed in his other questions.”

“Highly useful. They allow the interviewee to offer unanticipated and often most valuable observations.”

Ps. Additional Reasons…

Although it didn’t make the top five, several researchers cited one other notable reason for valuing open-ended questions, summarized in the following comment:

“They provide the rich unaided insights that often are the most interesting to our clients”

Next Steps: How to Get Value from Open-Ended Questions

I think we’ve established that most researchers recognize the tremendous value of feedback from open-ended questions and the reasons why, but there’s more to be said on the subject.

Conducting good research takes knowledge and skill. I’ve spent the last decade working with unstructured data and will be among the first to admit that while the quality of tools to tackle this data have radically improved, understanding what kind of analysis to undertake, or how to better ask the questions are just as important as the technology.

Sadly many researchers and just about all text analytics firms I’ve run into understand very little about these more explicit techniques in how to actually collect better data.