A NSW Government website

NSW Government
Digital NSW

Usability testing

Usability testing is a research method performed as part of the iterative design process to help you uncover problems or opportunities in the design of your product or service, and understand user's intentions when interacting with it.

What is it?

Usability testing is a method of testing the functionality of a service (for example, a website, app or other digital product) by observing real users as they attempt to complete tasks on it.

Why usability test?

The goals of usability testing vary by study, but they usually include:

  • Identifying problems in the design of the product or service
  • Uncovering opportunities to improve
  • Learning about the target user's behaviour and preferences.

It's important to do usability testing as part of your iterative design process so that any design decisions about your product or service are based on observations of real users, and of their interactions with the design.

Types of usability testing

There are different types of usability testing, but the core elements in most usability tests are the facilitator, the tasks, and the participant.

  • The facilitator guides the participant through the test process, working to ensure that the test results in high-quality, valid data without accidentally influencing the participant's behaviour. They give instructions, answers the participant's questions, and ask follow-up questions.
  • The tasks in a usability test are realistic activities that the participant might perform in real life. They can be very specific or very open-ended, depending on the research questions and the type of usability testing.
  • The participant should be a realistic user of the product or service being studied. They might already be using the product or service in real life or might just have a similar background or needs to the target user group.

Qualitative vs. Quantitative

Usability testing can be either qualitative or quantitative.

Qualitative usability testing focuses on collecting insights, findings, and anecdotes about how people use the product or service. Qualitative usability testing is best for discovering problems in the user experience. This form of usability testing is more common than quantitative usability testing.

Quantitative usability testing focuses on collecting metrics that describe the user experience. Two of the metrics most commonly collected in quantitative usability testing are task success and time on task. Quantitative usability testing is best for collecting benchmarks.

Moderated vs unmoderated

In a moderated test session, a moderator asks the participant to talk out loud while completing the tasks. In a separate space, the team watches the session and takes notes via video feed or through a one-way window. A moderator is present throughout the session to guide the participant on what needs to be tested.

You can perform remote moderated tests on Skype or Microsoft Teams where the facilitator and participant are in different physical locations.

An unmoderated session is a series of tasks that are inputted into a testing system ahead of time. Then the participant is asked to go into the system to complete the tasks on their own. Although it is more convenient to plan compared to moderated testing, the downside of unmoderated testing is that if a participant gets stuck or does something unexpected, you won't be able to probe in real time.

Number of users to test with

The number of participants needed for a usability test varies depending on the type of study. The objective is to test up to a point where you can identify trends and patterns and further tests will not give you much new insight moving forward.

For a typical qualitative usability study of a single user group, we recommend using five participants to uncover the majority of the most common problems in the product.

It's important to ensure you test with at least one user with physical or cognitive limitations or who needs assistance with digital applications. Testing with real users can uncover accessibility barriers better than a checklist.

Planning your tests

Before planning any sessions, work with your team to consider:

  • Research goals and tasks – you need to design test tasks to make sure they answer your research questions.
  • A prototype - the purpose of usability testing is to put something in front of the user and watch them work. This could be a prototype, working website, physical products.
  • Test participants – these need to be actual or likely users of your service.
  • Roles and responsibilities - each member participating in user research will have specific roles and responsibilities.
  • Time – you need time to plan the usability test. Budget in time for test preparation as well as running tests, analysing the data and presenting the findings.
  • Costs - consider how or where you will recruit your participant's and if you will be compensating them for their time or travel.
  • Test location – you don't need a formal lab. You can use meeting rooms, run pop up sessions or test remotely.

From testing to actionable insights

Usability testing is not just a milestone to be checked off on the project schedule. The team should have a goal for why they are testing and then implement the results. After conducting your research, analyse the data so you can produce useful insights that will help you design your service.

Resources

This content is based on work by the Ontario Digital Service (licensed under the Creative Commons Attribution 4.0 Licence).