Determining the right number of users for User Research.

Most clients and projects require the design researchers to state a predetermined sample size of users. Researchers often find it hard to justify to clients and business stakeholders on the choice of their sample size of users. Suggesting a very small sample size may make the client think that the research recommendations might not be impactful enough whereas, a very large sample size inflates the time and budget for the study. One of the many reasons for this conundrum may be that design research is often confused with market research and hence often assessed with the standards of market research. This report by Human Factors International superbly explains how they are different.

Well, theoretically a design research study can involve a minimum of 3 users to around 100+ users. Undoubtedly, the quality of insights drawn from such a range of users might differ nevertheless there is a combination of factors I believe one should consider when ascertaining the number of users. Let’s have a look at these factors in detail.

The Scope of the Design Study

The most important factor to consider is to determine what are we trying to accomplish through the study. The scope of the study can be:

a. To design a product from scratch and doing exploratory design research

When the focus is on discovering needs and opportunities for a product then, we need to maximize the number of users. This is because exploratory studies require to examine the entire spectrum of the target audience who might be existing or potential users of the product and also all possible behaviors, processes and interactions of these users with their operable environment which can lead to the creation of a new experience. Creating new experiences is like a gamble researchers and designers are taking. One can never be 100% assured that it’s going to work. Hence, the more the number of users, the better will be the statistical evidence to support the design hypotheses and ensure that the new experience is indeed a delightful one. However, the limiting factor of budget and time for the study can again constrain and influence the number of users for the study which I have covered later in the article.

b. To find solutions to problems of an already existing product and doing validatory research, eg. usability testing, identifying new features etc.

When we are finding potential solutions to difficulties with current design system then, we can go with a fewer number of participants. When probing for usability problems with an interface, sometimes the magic number of 5 users can do the trick as explained by this NN group article. However, when probing for new feature capabilities or studying the current state of the system then, sometimes one may need more than 5 users. In such cases, the decisions boil down to considering other factors such as characteristics of the study population, resources for the study, etc which I have discussed below..

Characteristics of the Study Population

It is uncommon to have just one kind of users of a product. Moreover, it is quite possible that the user groups may not be predefined and the research may further explore the various user categories in the study population. It is also possible that the target population might comprise users from various countries, cultures and even speaking many languages. Therefore, the more diversified a study population we are examining, the more the number of users will be required to get a comprehensive picture through the study. In other words, we will need multiple user representatives for each persona group or user type we are designing for. I recommend beginning with a minimum of three users to consider for each user type and then increase the number of users uniformly for each category depending on product and domain complexity.

Resources at Disposal

In reality, budget and time always dictate the sample size for research. In addition to conducting a research study, a researcher needs to budget time for recruiting users, analysing the data and further feeding the insights to design sprints. Hence, discerning the right number of users in perspective with time and budget of study is critical to ensuring complete project handling.

Considering the worst case scenario where the project permits very less budget and very less time, I would recommend to scale down the number of participants to get the data on time. This is to make sure that we are able to generate at least the most significant insights.

In cases, where there is ample time but less budget, we can always select research methods which are frugal enough to generate meaningful insights without compromising on the number of users being considered for study eg. methods such as design surveys, diary studies etc.

For the reverse case of less time and a substantial budget, we need to consider differentiating between exploratory and validictory research and further ascertain the number of users. This is because recruiting more users will not be a problem here, however being able to recruit more users in less time will definitely be.

Quantitative vs Qualitative studies

Out of all the factors which impact the number of users in a study, the choice of doing a quantitative study or qualitative study is the one 90% of the times in the hands of the researcher. This is because the purpose of research and the quality and type of insights which are expected from the study are most clear to the researcher. Rest all the factors are conditional and largely statutory to the project requirements. Thus, the researcher has the liberty of determining the number of users to fit its requirements. However, the only constraint to be careful here is findings from quantitative studies need to be statistically sound. Statistically sound quantitative studies set a requirement of a minimum number of users which is usually large (often 30 users or more). Whereas, findings from qualitative studies are behavioral estimates. Thus, the researcher has the freedom to decide the number of users which is further elaborated by this article.

To conclude

Small, medium or large — what sample size of users fits your study is a composite question. The magic number of 5 users may work magic in some studies while in some it may not. It depends on the constraints put on by project requirements, assumptions about problem discoverability and implications to the design process. Assess these factors to determine the number of users for your study:

1. What’s the nature and scope of research — is it exploratory or validatory?
2. Who and what kind of users are you planning to study?
3. What’s the budget and time to finish the study?
4. Does your research involve presenting statistically significant numbers or inferring behavioural estimates for the problem statement?

Happy User Recruiting!

Written by: Shrut Kirti (via Medium)
Posted by: Situated Research