Survey Tip: Go Easy on the Essay Questions

When is "outside the box" a bad thing? Answer: when it ruins your survey. And when it comes to information gathering, an essay question can be a real box-tipper.

Say you're conducting a customer satisfaction survey, trying to capture information about the percentage of employees or customers that have solved a work-related problem with the help of the IT office or your help desk.

You might want to focus their interaction with your survey on details of when, why, how long it took, and whether the problem resurfaced after the fix.

You could pose questions like this:

1. The IT/Help Desk responded to my request in the following timeframe:

  • Same day
  • 1 day later
  • 2–3 days later
  • More than 3 days later
  • Did not respond

Or you could pose a question like this:

1. Have you used the IT department/Help Desk to solve a work-related/product-related problem? Briefly describe the event and its resolution.

See the difference?

The first example is about a spectrum of results that can be quantified. The second stands to produce a soliloquy on anything from phone manners to personal hygiene. It only might bring in hard data about when, why, how long, and to what effect. Your busiest colleagues/customers might even default to "No problem. Worked out fine."

The point is: writing out sentences is perceptually a more complicated process than checking off boxes. Load your survey with empty space, and you may get them back with empty space. A Colorado State University guide to designing survey questions puts it this way: fight fatigue.

Another recent study of survey-question design out of the University of Washington (PDF) helps iterate an even more critical bottom line: measurable data makes for successful campaigns.

So, define your objective(s) and ask questions that provide the data to get there. When you open the playing field to the essay answer, you stand to lose control of that process. You've got to do the harder work of breaking up the larger question into component parts, and designing a non-essay questionnaire that steers your respondents into a place where they can easily share their experiences is the most data-centric way to do that.

This isn't to suggest that all essay questions have to go.

It's probably occurred to you already, so at the risk of pointing out the obvious: thoughtful inclusion of the essay question into a survey can provide a value-added component without detracting from the measurable data that you gather.

Think of the essay prompt in the example above as an "easy" question to include at the end of the survey. Once your respondents have sorted out all the multiple-choice items, they can add some anecdotal data to the quantifiable information they've provided. Or they may not, but your survey results won't be skewed by their exasperation at having to compose a bit of prose, when what they really want to do is outline their experience quickly and effectively.

Let 'em work for you. Save the essays, use them sparingly, and keep them for the very last question.

by Josh Pigford