Tips when categorizing text answers

Open-ended questions are the best way to gain new knowledge from your respondents as they can write whatever they want instead of picking from a set list of options. For that reason, open-ended questions are great for follow-ups such as "Other ..." or when you want opinions and suggestions. Despite the benefits, open-ended questions should be used sparingly in web surveys as they require more of your respondents' time and when overused can cause them to leave the survey halfway through. The text answers from open-ended questions are also more difficult to analyze as you'll just get a long list of answers, but that's where the categorization comes in.

In this article, we'll go through why to categorize, how to identify categories, how to name categories, general tips, and tips when categorizing many answers.

Why should you categorize the text answers?

The main reason to categorize text answers is to get at better overview of what the respondents are saying. When you categorize, you group similar answers together which allows you to focus on the categories instead of individual answers. The categories can also be presented as a bar chart, which is easier to understand, present, and share with colleagues.

Besides a better overview, categorizing the answers helps you identify problems with the survey and the website, but also determine which comments to listen to. When reading the text answers, individual comments can be convincing and it's easy to forget that it's still the opinion of just one person. However, if you categorize, you get a better understand of the general opinion and what you should prioritize.

How to identify a category

When you categorize the answers for the first time, it's important to start with an open mind. You don't want to pre-define the categories and force the answers to fit the categories. Instead, you should start by reading through the answers and see what the respondents are saying. You'll soon start to notice patterns. When you have noticed several similar answers, you can create a category that matches and add those answers to the category.

There are no rules for what defines a category. Instead, it will depend on the answers and the information that you and your colleagues need. For example, if it's relevant to separate different professions, use one category for "journalists" and another for "editors", but if that's not relevant you can group both professions under the umbrella "media". It's generally better to aim for fewer and broader categories instead of lots of small categories. If you're too precise, the list of categories will almost be as long as the list of answers, and the categorization no longer serves a purpose.

How to name the categories

As mentioned before, there are no rules you have to follow, but to make the data analysis easier, here are some suggestions for how you can approach the categories for different types of questions.

Follow-up questions (for example "Other role" and "Other purpose")

You'll be surprised how often respondents will write an answer for the follow-up question that was already represented as an option in the main question. For this reason, it's good to use the option names as your category names when relevant. For example, if one of the options for role is "Potential customer" and you have text answers for "Other role" suggesting they want to purchase from you, you should categorize them as "Potential customers". This will make it easier to see if respondents are misinterpreting the options, or perhaps if they are just not reading carefully (maybe there are too many options, making the respondents skim the list).

If the answers are not matching one of the options for the main question, group the answers in a way that makes sense to you and how you work with the website. For roles, you can single out the interesting roles in smaller categories and group the less interesting roles in bigger categories. When possible, it's best to use categories that could be options in the main questions. Otherwise the conclusions from the main and the follow-up questions may not match.

Purpose

For text answers related to the visitor's purpose and reason for visit, it's best to name your categories in accordance with the existing headings and actions on the website. For example, the categories can be "Log in", "Sign up for newsletter", "Find product", or "Contact information". By matching the website, it'll be easier to analyze where the visitors are struggling, but also connect the results with your website goals and user flows.

Cannot find

What the visitors cannot find is related to their purpose, and you should therefore try to use the same categories as much as possible. Besides the benefits from the previous section, it will also make is easier to see the relationship between the results for the different question.

Improvement suggestions

There is often a direct relationship between what the visitor was struggling with (cannot find) and what they would like you to improve, but that's not always the case. Sometimes they are happy with the experience but still think the design could be better, or they have a specific issue but can't translate that into a concrete suggestion. When possible, it's best to use the same categories for what the visitors cannot find or do and what they want you to improve, but you will most likely have to add more categories for the suggestions too. Common categories are "Navigation", "Search engine", and "More information". Again, what is most important is that the categories align with your work so that the suggestions can lead to actual improvements.

General tips

When to add new options to the main question

One benefit of text answers is that they can help you review your survey questions. If a lot of your respondents choose "Other" instead of any of the other options, that usually indicates that the question is missing an option or two. However, you need to use a bit of math before determining which new options to add. Even though 35% of the respondents answering the follow-up question said the same thing, that may only represent about 1% of all the respondents. In that case, the option is usually to small to add to the main question. Try to aim for at least 3%. Exceptions can be made for options that are important to you, such as a role or a purpose you want to focus on for your website.

Avoid categories that are too narrow

If you use very narrow categories, you will end up with only a few answer per categories and the analysis will only be slightly improved from not having any categories at all. Instead of naming categories based on the exact answer a respondent wrote, such as "looking for a welding position", you should focus on the overarching theme, such as "job seeker". It's a lot easier to draw conclusions when you have bigger categories.

Avoid categories that are too general

Although the categories shouldn't be too narrow, they can also be too general. Where to draw the line will depend on the question, but when a category is too vague it no longer adds anything to the analysis. Vague categories will often contain diverse information.

If you notice that a category is being used very frequently, it's time to see if you can identify any sub categories. For example, if most the respondents are commenting on the navigation, there may be a pattern where some are commenting on the main menu, some on the internal links, and some on the search engine. In that case, those are better categories than the overarching "Navigation".

Avoid hierarchical categories

This tip only applies when working in a tool that is not built for hierarchical categories.

When using hierarchical categories in a flat structure, it will be difficult to remember how you've structured the categories. It will be even more difficult to understand for your colleagues. The problem with hierarchies is that the same content in a text answer will match more than one category. Should you choose "Main menu" or "Navigation"? It's therefore better to keep the categories on the same level, but you can have a category for "Other navigation issues" if there are some answers that don't match the more specific categories.

You don't have to categorize every answer

It's tempting to want to add a category to every answer, but that will often be pointless. As mentioned before, you don't want very narrow categories, but if there are no bigger categories that match, then you'd have to use a very general category, which you shouldn't do either. Instead, learn to be comfortable with leaving some answers without a category. The point of categorizing is to find the main themes, not every theme.

Also categorize non-answers

For text answers, you will always get non-answers, such as "n/a", "don't know", "no", or "-". One way to reduce the frequency of these answers is to always have your open-ended questions be optional. However, you will still get answers likes these and you should have a category for them. There are two reasons for this. The first is that it will reduce the number of answers without categories and help you see that you've gone through all the answers. It's just more satisfying if the bar chart of the categories adds up to almost 100%.

The second reason to categorize non-answers is that this too can be an indication that you should tweak the survey. If a high share of the respondents avoid answering the question, you should probably change the question to increase the response rate. Perhaps they don't understand, feel it's too personal, or don't feel like writing a long explanation. It can also indicate that the respondents feel that they've already given you the answer to this question earlier in the survey, and in this case you should probably remove the question instead.

Don't feel limited by your own categories

Only you will know which categories are relevant to you and your website, and this may also change from year to year. Don't feel you need to stick to the categories you create the first time you categorized answers. If a categories isn't relevant anymore, don't use it. If you want a new categories, add a new category. This is simply a tool to help with the analysis.

Tips when categorizing many answers

If you have a lot of visitors on your website or if you rarely categorize your text answers, you may end up with hundreds or even thousands of uncategorized text answers. It can feel a bit overwhelming to start, but here are some tips to help you. These tips are written for the Extellio tool but can often be applied in other tools too.

Please note that in our tool, you can only view the first 1000 text answers for a selected time period. If there are more than 1000 text answers for the date range you want, you need to look at shorter date ranges to cover all answers.

Search for key words

In the categorization tool, there is a search function where you can search for words in the answers. If you notice that there are commonly used words, such as "consultant" or "yearly report" or "diet", you can search for those words to filter the answers. Now you're able to add a category to all the selected answers instead of going through them one by one. To do so, select the suitable category in the "Select tag" field in the top left corner. With the tag selected, you can use the button "Add to X responses". 

This way of categorizing is a lot quicker but less precise. You should therefore only use this method for words that are unique for that category. However, you can also skim through the answers and see if you need to change the category for any of them.

Filter alphabetically

When categorizing answers individually, a way to pick up the pace is to filter the answers alphabetically. You do that by clicking on the table title "Text" in the tool. By having the answers in alphabetical order, you often end up with similar answers after each other, which means less time looking for the appropriate category. This usually makes the categorization quicker, though you'll likely notice that this method will be more effective for some of the answers than others.