One of the most common questions asked on our courses is how many users are needed for a usability test.
It’s something that people are unsure about – how many is too many? What is the minimum amount of people you need for successful testing? And so on…
Often companies overestimate how many users they need and that completely puts them off doing usability testing in the first place. When, actually, it’s easy to do usability testing with a small enough group of users – it’s very cost effective and delivers huge insights into user needs and behaviours.
With that in mind, here’s our guide to how many users you need for usability testing.
Don’t overestimate how many users you need for a successful usability test
“One of the reasons why people overestimate how many users they need is that they aren’t clear on the difference between quantitative and qualitative research,” says Colman Walsh, founder of the UX Design Institute.
“They think all research is the same and when they think of research samples, they think of quantitative research. Typically that’s when you sample 1,000 people for an opinion poll. So you might think that you need to do usability testing like that. But, in fact, usability testing is qualitative research.
“What you’re looking for is not statistics or proof of anything – you’re looking for insights into user behaviour.”
How many users should you use? And why?
“Typically, we would always suggest five or less,” says Colman. “And never more than five, except in certain situations. Five is a good number for lots of reasons. I don’t know why I hit on five – I suspect I did because of Jackob Nielsen’s recommendations but over the years, I’ve stuck on five and there are many good reasons for it.”
Jackob Nielsen of Nielsen Norman Group sets out his case for using no more than five users in this blog.
He says, “The best results come from testing no more than five users and running as many small tests as you can afford.”
So why is five users the magic number?
“You can get your round of user testing done with five users in one day, so it minimises the amount of time you need to invest,” says Colman. “If you think about it – each user takes an hour, that allows you do five tests in a day with at least half an hour break in between. You do need that half hour break to gather your thoughts, take notes and so on.
“There’s also just a case of diminishing returns. No matter what you’re testing and how many users that app or software might have, usually after three or four users and definitely after five, you start to see the same usability issues crop up again and again.
“What that has told me is that you uncover the vast majority of usability issues with five people and when you start going beyond five, you’re really not finding out anything new. So there’s no point really.”
Steve Krug agrees that there are diminishing returns once you go beyond a certain number of users (in his case, he specifies three as being the ideal number of users).
In Don’t Make Me Think, he writes:
“The first three users are very likely to encounter nearly all of the most significant problems, and it’s much more important to do more rounds of testing than to wring everything you can out of each round. Testing only three users helps ensure that you will do another round soon.
“Also, since you will have fixed the problems you uncovered in the first round, in the next round it’s likely that all three users will uncover a new set of problems, since they won’t be getting stuck on the first round of problems.”
Five users gives you some wiggle room
It happens often enough – you have five users booked and someone drops out at the last minute. If that happens, you’ve still got four for your usability testing. But if you’ve only gone for three users in the beginning and someone drops out, you’re stuck with two.
“Testing with two would mean you’re wondering, ‘Did we really uncover everything we needed to uncover?’ and ‘Were these two people outliers in any way?’.” says Colman. “Usually with five people in a usability test, you’ll uncover any outliers.”
“Steve Krug talks about three users but I think it could be a bit risky because you might not uncover everything. And if you’re going to a client or stakeholder to say you’ve tested this many people, there’s a bit more weight behind it if there’s five versus three. Plus, if someone drops out or you have a participant who, for whatever reason, isn’t useful when it comes gathering insights, I think five give you that extra bit of wiggle room.”
Testing a small group of users helps with clearer reporting
Steve Krug points out that testing a smaller number of users, “makes it possible to test and debrief in the same day, so you can take advantage of what you’ve learned right away. Also when you test more than four at a time, you usually end up with more notes than anyone has time to process – many of them about things that are really ‘nits’, which can actually make it harder to see the forest for the trees.”
Colman agrees that having a smaller group of users helps make report writing more concise.
“What we found worked really well with our clients is less of a report actually and more video footage. Because seeing is believing. Capturing the usability testing on video really hammers home the message that there were problems and what the problems were.
“Following up the video footage with a fairly light report without too much commentary is the way to go – the report outlines the problems and suggestions for how you might address them.
“I’d say on a scale of 1 to 10, video footage has an impact of 10 out of 10 and a standalone report has an impact of 2 out of 10. Anybody that does usability testing and doesn’t record the tests to compile a highlight reel is missing the biggest opportunity and benefit of doing usability testing. It’s as much about identifying usability issues as it is about persuading people that there are usability issues.”
Can you have too many participants in a usability test?
“Yes, I think you can,” says Colman. “Once you get into six, seven, you’re wasting your time. You’re adding more analysis and video compilation time and not seeing any additional benefits.
“But there are exceptions to the rule – when you might want to use more than five users. If you have a website with lots of functionality – say an airline website – you’ll probably want to test two distinct pieces of functionality. Let’s say you want to test the purchase process and the check-in process. Instead of trying to cover both those things in one test, you can have two separate tests – one is on the purchase process with five users and one is the check-in process with five users. So it’s more tests rather than more participants in each test.
“Some websites may have completely different user types. I worked for property website Daft and there were users buying a property, renting a property, selling a property or they could be a landlord renting out a property. And they could have completely different behaviours, using different functionality, different tools, different parts of the software.
“So instead of doing one test with 20 different participants, we’d do one test with five landlords, another test with five renters, one with five buyers and so on. What it means is you’re doing more tests but not trying to jam more participants into one test.
“One final reason to have more than five users is for political reasons. You might have someone higher up the food chain who says they want statistical proof coming out of these usability tests.
“It isn’t the right approach for qualitative research but someone might just insist that you need to go up to 20 users to show that 12 out of 20 people had difficulty creating a password or 17 out of 20 had problems logging in, or whatever it might be. It’s kind of mixing apples and oranges to use a qualitative technique to generate quantitative findings but sometimes it has to be done.
“But only really do it if somebody higher up the food chain didn’t really understand the difference between quantitative and qualitative and felt they wanted this kind of proof and you weren’t able to persuade them. It can happen sometimes that you can’t persuade them – it happens to even the most experienced of us.”
Why having five users is more cost effective than you might think
“If you’re testing five users, it’s very cost effective – when you think about the insights that you get at the end of the testing. It costs €100 per person to recruit users (using a research recruitment company) and you give an incentive of €50 to each user. Combined with inexpensive recording software for the testing, it’s going to cost you roughly €900.
“Once you go beyond five users, it’s an extra day, more time and therefore the cost increases. Whenever I’ve done usability testing for clients, whatever we did, we’d always recommend five users. It’s the right number for research insights and it’s cost effective for the client too.”
Where can I find users for a usability test?
Now that you know how many users for a usability test is appropriate, check out How to find users for a usability test.