November 18, 2005
Bias in Usability Testing - Carolyn Snyder speaks to PhiCHI
Carolyn Snyder (author of Paper Prototyping gave an interesting presentation to the PhiCHI/Philly CHI group on Monday night. She covered a whole range of examples, from obvious problems with the facilitator and tasks, to more subtle issues arising from the testing methodology and the analysis and reporting of the test data.
For example, the commonly-used “think aloud” methodology is not normal behavior for users and can be awkward, slow, and draw attention to items that aren’t major problems. (It occurred to me that even thinking is often not normal and this method also forces users to consider each action to a much greater extent than normal.)
I picked up a great idea for making a last ditch effort to address concerns about bias that arise after testing. Snyder suggested assembling a qualitative analysis, a document listing all of the possible biases affecting the results and the degree of each item’s impact. This helps to put things into perspective and ensure that everyone is considering the complete picture.