One-on-one interviews with users (use these same techniques for Focus Groups)
Protocol testing will tell you what a user thinks a document means and will help you determine if the user is interpreting your message as you intended.
Try to conduct 6 to 9 interviews on each document. Ask the tester to read to a specific stopping point, known as a cue. Each time the tester reaches a cue, ask for an explanation of what that section means. At the end of the document, ask additional, open-ended questions, such as
- What would you do if you got this document?
- Do you think the writer was trying to help you?
- Do you think your friends would understand this document?
This last question is important because sometimes people are more comfortable telling you what they think others might find confusing, rather than admitting that they don't understand something themselves.
Don't ask yes/no questions. You won't get much usable information from that type of question.
Use a different type of protocol testing when evaluating long documents, like booklets or regulations. You may know this type of testing as Usability Testing. In this testing, not only do you test for comprehension, but you also make notes about the way the user uses the document. For instance, you note how often a reader has to flip from page to page to find references. You test the document as a whole, not just individual paragraphs.
Protocol testing is time consuming, but the time invested is worth it. Taking the time to test your document up front may save you hundreds of hours later answering questions from your users or producing a second document clarifying the first one.
The information was so general that it would have generated calls:
Veterans Benefits Administration tested a letter in which users appeared to understand every word. However, when asked what they would do if they got this letter, most people said they would call VBA's toll-free number.
The letter was about a replacement check sent because the original check was out of date. The letter said, "You will receive the new check shortly." Readers indicated that they would call if they didn't receive the check at the same time as the letter. Changing the sentence to show an approximate date they would receive the check eliminated countless phone calls.
A "term of art" that VBA thought veterans understand would have caused readers to take the wrong action:
When testing a multi-use letter, some readers were confused by the term "service-connected disability." To VBA it means that a veteran has a disability that can be traced back to time in military service." Protocol tests showed that one veteran thought it meant a disability that happened at work. Another thought it meant you had to be injured while in the military, but not necessarily while on duty. Another thought you had to have gotten the disability during combat for it to be considered service-connected.
When each reader was asked a general question about understanding the letter, they all said that it was clear. Yet several would have done something other than what VBA wanted because they had a different definition of "service-connected." To solve this problem, VBA explained the phrase so that everyone was working from the same definition.
Adding a word to make something more legally sufficient would have caused readers to give incorrect information:
A team working on a form wanted to use the question, "When were you last (gainfully) employed?" They felt that the term "gainfully employed" would gather more legally sufficient and accurate information than just the word "employed."
Testing showed that readers used at least three different definitions of "gainful" employment:
- Any job
- A job that provides benefits or where you can put money away
- A job that keeps you above poverty level
In fact, research showed that different government agencies may have different definitions of "gainful." But, more importantly, because each reader had a different definition of the word, the agency would have gotten less accurate information if the word had been in the document.
Remember, the goal of protocol testing is to ensure that your audience understands your document, and therefore, won't have to call you for an explanation. Although this technique is very valuable, it probably isn't worth the time to test documents that go to only one or a very few people.