InteractionArchitect.com
Observation methods and tips for usability testing


There are two observation methods for usability testing. "Unobtrusive observation" means you observe what test users do and refrain from interacting with them. With unobtrusive observation you learn whether the system is easy to use. "Obtrusive observation" means you interact with test users, e.g. by asking questions. With obtrusive observation you learn more about the usefulness and acceptance of the system. Besides explaining these observation methods, this article provides practical observation tips for each method.

Introduction: what is usability testing?

Usability testing is a technique to evaluate the ease of use or ease of learning of an interactive system. During a usability test a real future user uses (a prototype of) the system while one or more observers look at how this is done. Usability tests are often task-oriented, i.e. the test user receives a set of predefined tasks to perform. Usability tests often use the think aloud technique, i.e. the test user is asked to say aloud what he thinks while using the system. With usability testing, you learn whether the design is adapted to future users and their tasks.

Two observation methods

There are essentially two observation methods you can use during usability testing. The first method is "unobtrusive observation": you concentrate on observing what the test user does and refrain as much as possible from influencing her/him by explaining the design or asking questions. This is harder than it sounds and it actually takes some tricks to do it well.

The second method is "obtrusive observation", which means you are allowed to explain design decisions, ask questions, or engage the test user in a discussion.

Neither method is right or wrong. Each uncovers different things and you will want to use both methods in one usability test to learn as much as possible about the usefulness and usability of your design. One way of doing that is to start each usability test with unobtrusive observation: you observe how users execute the tasks you give them. After that, you reserve some time to ask questions, explain design decisions, and answer the test user's questions.

Unobtrusive observation

With unobtrusive observation you learn whether people can use your design in an easy and efficient way, and where this is not the case. How people behave and how people explain their behavior are two different things. If you want to learn about behavior, you have to study behavior. If you want to learn whether users can use your design, you have to observe how they use it, not ask them what they think of it.

Opinions are bad predictors of behavior. During one usability tests we observed that test users were able to complete only 1 out of 5 basic tasks with the system. But when we asked them their opinion afterwards, they told us the system was great.

  Tips for unobtrusive observation

1. Observe: be quiet, watch, understand.
  • Don't explain
  • Don't ask the test user's opinion
  • Don't defend the design
  • Don't apologize
  • Don't suggest
  • Don't contradict the test user nor agree with him/her: stay neutral
This is actually very hard. It seems to be in our nature to help others when they are in trouble. It is even more difficult to see test users struggle with the design if you have worked on the design yourself. Just keep in mind that you're there to learn about the test user's behavior in relation to your design, not to convince or teach the user.

2. Only help to overcome the limitations of the prototype. Explain briefly and in a neutral way what would happen in the future system.

3. First observe, then take notes. Don't let your note taking get in the way of observing what the test user is doing. You don't have to write down everything you notice during observation. Instead, take 15' to clean up and complete your notes after the observation session. But don't postpone it because you will quickly forget important details.

4. Stimulate users to think aloud. But use neutral prompts, e.g. “What do you see, what are you thinking, what do you want to do, what are you looking for?”

5. Limit the time test users have to execute a task. Don't prolong the test user's suffering longer than necessary. If a test user is really stuck on a task and you have learned why this is the case, thank the user for trying and ask him/her to continue. Usability testing can be a frustrating experience for test users because people seem to be naturally inclined to do the best they can.

6. Elicit detailed information.

Test User: “I see a lot of information.”
Observer: “Could you tell me what information you see?”

7. Answer test user questions with questions. Dealing with test user questions is probably the most difficult aspect of unobtrusive observation. It seems natural and polite to answer questions. Before each usability test, you should explain the test user how you will deal with questions. You should encourage the test user to ask questions because that allows you to learn what is not clear in the system. Explain that you will not answer these questions right away but that you will write them down and answer them at the end of the test.

A productive way of dealing with test user questions is to answer them with questions. If you simply leave the question unanswered, the test user might feel ignored. His/her motivation to think aloud and to continue asking questions will diminish or even vanish. By bouncing the question back you acknowledge the question and you encourage the test user to elaborate his/her question, enriching your observation. To minimize your influence on the test user, use the same words as the test user.

Test User: “What does this text mean?”
Observer: “What do you think it means?”

Test User: “Do I have to click here?”
Observer: “What do you think will happen if you click there?”

Obtrusive observation

What unobtrusive methods will not tell you, is what test users think of your design. Do they like to use it? Does it answer their needs? You can't observe opinions by just watching people. Even extreme emotions are difficult to observe accurately: does my test user get angry because s/he doesn't understand my design or because s/he thinks it is so simple it insults her/his intelligence.

If you want to learn more about the usefulness and the acceptance of your design, you will have to ask test users.

  Tips for obtrusive observation

1. Think about what you want to ask before the test. When you have just finished a test session with unobtrusive observation, you will have a lot to talk about with the test user. You can ask the user to clarify or explain actions you have observed, you can explain and discuss design decisions, etc.

But you will also need a checklist of things you want to know from all test users. This checklist will contain items related to functionality (what is useful - useless - missing?) and items related to user acceptance (what do they like - don't like).

2. Ask open questions. Avoid closed questions, i.e. questions that can be answered with yes or no. You will get more detailed and accurate information with open questions.

Bad: “Do you understand what this means?”
Good: “What do you think when you see this?”

Bad: "Did you know you can click here to achieve that?"
Good: "What would you do if you would want to achieve that?"

3. Don't blame the test user. Remember: you're testing the usability of the design, not the computer literacy of the test user. If the test user does not understand something, this something will have to be improved in the design.

Bad: “Why don't you understand this?”
Good: “Could you tell me what this means for you?”

4. Don't ask the test user for design solutions. Test users are not interaction designers. That is why test users will almost never provide you with good design solutions. Don't bother asking, but take note of suggestions they make spontaneously. It is your task to design new interaction solutions on the basis of your better understanding of user needs.

Bad: “Do you need a News button here?”
Good: “Which information do you need at this point?”

Instead of asking for design solutions, it is sometimes useful to ask the test user to compare the system with related products s/he has used before. By using interactive systems everybody gains tacit knowledge of interaction design. But because test users are not interaction designers, they lack the vocabulary to make this tacit knowledge explicit. It is much easier to compare the system with actual experience with a related product.

Bad: "How can we redesign this page to make it easier for you?"
Good: "Compared to the product you have used before, what do you think is better or worse in this system?"

5. You can do obtrusive observation with groups of test users. People often find it easier to formulate their opinions when they are confronted with the opinions of others. That is why you learn a lot about the usefulness and acceptance of your design when you ask a group of test users to discuss your design. Your role is to facilitate and focus the discussion.

Group discussions are harder to organize than individual sessions because you need several test users at the same place at the same time. They are also harder to manage because you have to control not only individual reactions but also group processes. You will want to stimulate some group processes (e.g. opinion disclosure) but at the same time avoid unwanted group effects (e.g. group intimidation).