Useability Evaluation Considered Harmful

comments: a, b, c

Summary:

This paper talking about how too much emphasis is put on useability evaluations. Useability evaluations can be very useful, but too often they are put together at the last minute, because even good ideas are considered bad if they don't have a useability study. This leads to useability studies that only use a handful of people, and those people could be close friends of the researchers leading to bias. In addition, there are potentially good ideas that do bad on useability studies because the researchers don't have time to implement a complete program utilizing the new interface. They usually have a prototype or "sketch" and so if it fails during an evaluation, it could be that the program was incomplete, not that the interface is bad.

My Thoughts:

I agree with this paper. In many fields outside CHI, it is the same way. They do studies that have many biases just to prove their point. Whenever I hear about a study that proves something, I never trust it unless I hear exactly how the study was performed. That's how CHI should be. If someone conducts a study, it shouldn't mean anything unless you look at how the study was perfomed. You also need to consider that biases are most likely involved. One great point the paper made was that studies are hardly ever repeated. A key point about using a study to prove something, is that you need to be able to repeat it and get the same results. Since people don't repeat studies, researches can get away with using a ton of biases in their usability studies.