Paraphrase testing from the Veterans Benefits Administration
The information was so general that it would have generated calls:
Veterans Benefits Administration tested a letter in which users appeared to understand every word. However, when asked what they would do if they got this letter, most people said they would call VBA's toll-free number.
The letter was about a replacement check sent because the original check was out of date. The letter said, "You will receive the new check shortly." Readers indicated that they would call if they didn't receive the check at the same time as the letter. Changing the sentence to show an approximate date they would receive the check eliminated countless phone calls.
A "term of art" that VBA thought veterans understand would have caused readers to take the wrong action:
When testing a multi-use letter, some readers were confused by the term "service-connected disability." To VBA it means that a veteran has a disability that can be traced back to time in military service." Protocol tests showed that one veteran thought it meant a disability that happened at work. Another thought it meant you had to be injured while in the military, but not necessarily while on duty. Another thought you had to have gotten the disability during combat for it to be considered service-connected.
When each reader was asked a general question about understanding the letter, they all said that it was clear. Yet several would have done something other than what VBA wanted because they had a different definition of "service-connected." To solve this problem, VBA explained the phrase so that everyone was working from the same definition.
Adding a word to make something more legally sufficient would have caused readers to give incorrect information:
A team working on a form wanted to use the question, "When were you last (gainfully) employed?" They felt that the term "gainfully employed" would gather more legally sufficient and accurate information than just the word "employed."
Testing showed that readers used at least three different definitions of "gainful" employment:
- Any job
- A job that provides benefits or where you can put money away
- A job that keeps you above poverty level
In fact, research showed that different government agencies may have different definitions of "gainful." But, more importantly, because each reader had a different definition of the word, the agency would have gotten less accurate information if the word had been in the document.
Remember, the goal of protocol testing is to ensure that your audience understands your document, and therefore, won't have to call you for an explanation. Although this technique is very valuable, it probably isn't worth the time to test documents that go to only one or a very few people.