One-on-one paraphrase testing sessions work best for short documents, web pages, and survey questions.
Paraphrase testing will tell you what a reader thinks a piece of writing means and will help you know if they’re interpreting the message as you intended.
Conduct between 6 to 9 interviews
Ask the participant to read to a specific stopping point, known as a cue. Each time the participant reaches a cue, ask the participant to tell you in his or her own words what that section means. Take notes, writing down the participant’s explanation in the participant’s words. Do not correct the participant. When you review your notes later, wherever participants misunderstood the message, the document has a problem that you should fix.
Ask additional, open-ended questions
- What would you do if you got this document?
- What do you think the writer was trying to do with this document?
Thinking of other people you know who might get this document:
- What about the document might work well for them?
- What about the document might cause them problems?
This last question is important because sometimes people are more comfortable telling you what they think others might find confusing, rather than admitting that they don’t understand something themselves.
Avoid yes or no questions
You won’t get much usable information from that type of question.
With only 6 to 9 participants, paraphrase testing will not take a lot of time, and the time invested is worth it. Taking the time to test your document and change it based on what you learn may save you hundreds of hours later answering questions from your users or producing a second document clarifying the first one.
For longer documents where finding information is also important do usability testing. Usability testing is the best technique for booklets, regulations, and web sites. With usability testing, you test the document as a whole, not just individual paragraphs.
Paraphrase testing from the Veterans Benefits Administration
Veterans Benefits Administration refers to paraphrase testing as protocol testing.
The information was so general that it would have generated calls:
Veterans Benefits Administration tested a letter in which users appeared to understand every word. However, when asked what they would do if they got this letter, most people said they would call VBA’s toll-free number.
The letter was about a replacement check sent because the original check was out of date. The letter said, “You will receive the new check shortly.” Readers indicated that they would call if they didn’t receive the check at the same time as the letter. Changing the sentence to show an approximate date they would receive the check eliminated countless phone calls.
A “term of art” that VBA thought veterans understand would have caused readers to take the wrong action:
When testing a multi-use letter, some readers were confused by the term “service-connected disability.” To VBA it means that a veteran has a disability that can be traced back to time in military service.” Protocol tests showed that one veteran thought it meant a disability that happened at work. Another thought it meant you had to be injured while in the military, but not necessarily while on duty. Another thought you had to have gotten the disability during combat for it to be considered service-connected.
When each reader was asked a general question about understanding the letter, they all said that it was clear. Yet several would have done something other than what VBA wanted because they had a different definition of “service-connected.” To solve this problem, VBA explained the phrase so that everyone was working from the same definition.
Adding a word to make something more legally sufficient would have caused readers to give incorrect information:
A team working on a form wanted to use the question, “When were you last (gainfully) employed?” They felt that the term “gainfully employed” would gather more legally sufficient and accurate information than just the word “employed.”
Testing showed that readers used at least three different definitions of “gainful” employment:
- Any job
- A job that provides benefits or where you can put money away
- A job that keeps you above poverty level
In fact, research showed that different government agencies may have different definitions of “gainful.” But, more importantly, because each reader had a different definition of the word, the agency would have gotten less accurate information if the word had been in the document.
Remember, the goal of testing is to ensure that your audience understands your document, and therefore, won’t have to call you for an explanation. Although this technique is very valuable, it probably isn’t worth the time to test documents that go to a small number of people.