Collecting data to measure performance
Collecting data about your service allows you to measure its performance. It is essential that you continually test and measure to ensure your service is meeting user needs.
You can use data to test if:
- The service is meeting user needs
- The service allows users to easily complete tasks
- There are enough people using the service to make it cost-efficient
- People know about the service and are using it.
What are metrics?
Metrics are measurements that tell you how well something is performing. Typically, they're expressed as a percentage. For example, if 2000 people tried to complete your service, that's a measurement of total attempts. If 1000 of them successfully completed your service, that's a measurement of completions. Combine the two and you get a metric for how often people using the service manage to complete it. In this example the completion rate is 50%.
Metrics to measure
Collect data that shows how your service is performing against these four metrics, also known as key performance indicators (KPIs):
- Cost per transaction - how much it costs the NSW Government each time someone completes the task your service provides
- User satisfaction - what percentage of users are satisfied with their experience using your service
- Completion rate - what percentage of transactions users successfully complete
- Digital take-up - what percentage of users choose your digital service to complete their task over non-digital channels
What to test
Depending on the type of service you are developing, you may also need to test:
- Error rates
- Time to completion
- Costs, benefits and return on investment
- Content metrics (readability, length)
Measuring in discovery and alpha
You must consider how you'll measure your service from the start of your project.
In discovery and early alpha, you need to:
- Have an analyst as part of your team (or available to your team) so you can start asking the right questions about how you're going to measure service performance
- Estimate the number of people you expect to use the service - be aware large numbers may mean you need powerful analytics tools
- Find out the analytics tools your organisation already has and whether they're suitable for the type and volume of data you're expecting
- Find out where existing data is kept and how you're going to access it, aggregate it and make it usable so that you can measure your KPIs.
Measuring in beta
By the end of beta you will be able to show:
- Which metrics and measurements you will use to monitor your KPIs
- Baseline measures and benchmarks for success
- That the team is ready to report their performance
- Which tools you use for analysis and web analytics in beta (and alpha if appropriate)
- What you have learned from qualitative and quantitative data, for example key evidence
- That during public beta phase you have been able to test your methods for data collection, validated the data is accurate, and published service performance data on a performance dashboard.
- As you go live you should be able to show service data on your performance dashboard and improvement to the service based on performance data.
Using data
Your data should show
- Increased user satisfaction
- Increased digital take-up
- Completion rate has been maintained
- Decreased cost per transaction.
How to use your data
It's important to use the data you've collected to find ways to improve your service.
For example, your completion rate data can help you identify the stages in your user journeys such as when users are dropping out. Work with your user researcher and plan a round of user research to understand why.
You can also segment user data based on the characteristics of groups of users. For example, you could check if users of a certain age find the service harder to complete than other users. If this is the case, you can work with your user researcher to plan a round of research with those users.