Keep these tips in mind when conducting internal user tests with colleagues

Testing with colleagues can provide value as a layer within your research and have positive consequences for communication within the company.

In part one of this article, I outlined some unexpected benefits from conducting user tests with colleagues. These include increased transparency, involving more people in the design and research process, increased empathy and understanding of usability topics, a refined research guideline, and reduced cost, increased sample size.

Still, there is a knee jerk reaction when mentioning internal user tests with colleagues. The resistance to testing internally with colleagues is likely rooted in a time when that is the only type of testing some companies did. In this case, it can be dangerous and lead to incestuous thinking patterns, but I would argue that we have gone to an extreme in avoiding testing with colleagues (or not talking about it if we are conducting internal user tests). I am not advocating for swinging the pendulum in the other direction, but it is time for us to have a more candid conversation about the benefits of testing with colleagues.

As I have mentioned, conducting user tests with colleagues should never be the only type of testing you do, but rather it can complement other methods and help you make the most of time you have with users. I think we need to change the topic from a taboo one (e.g. we should never test with colleagues) to an acknowledgement that there is a time and place for testing with colleagues, and that this might come with added advantages, both for the product and communication within the company. Below are some things to keep in mind when thinking about running internal user tests with colleagues.

Try to limit familiarity with the site

• The prototypes we have been testing with colleagues are a new design of the website, so an added advantage in this situation is that colleagues are not familiar with the designs. That is to say, I am not testing designs of the website that everyone has known for years, but rather new content that most colleagues have not seen before. Still, colleagues inevitably compare the new designs are then compared with the current website, but the same can be said of users. Regardless of whether you are testing big design changes or small tweaks, it is always best done with colleagues who are not involved in the product teams. Consider testing with those not directly involved in building the product, like sales, human resources, and administrative staff. Another approach to limit bias due to company knowledge is to recruit new hires who do not yet know the website in and out.

Focus on broad usability questions

• Similarly, the questions I often have for internal user tests were quite broad in nature (e.g. Can the Menu be easily found?). If the questions are very specific to a target group, testing with colleagues might not be the best way forward, and in fact, can be misleading. Testing internally might be more relevant for situation when you are trying to understand if a button is discoverable, than situations where you try to find needs specific to your target audience. There are questions that need to be answered by your specific target group, and there are broad usability questions that need to answered by humans.

Interpret results in light of potential biases

• I am currently working at a company that sells (mostly) package trips. Our average user is likely older than our average employee, and of course, less computer literate. Think critically about how your colleagues are different compared to your target audience and make sure to consider these factors in your analysis and interpretation. You might say that you run a first round of tests with colleagues, and then a second iteration with users who are on one end of the spectrum in order to test if the design suits their need and is accessible to them. Ask yourself: How different are your colleagues compared to your target audience? What bias does this lead to? Make sure to keep this information in the analysis and communication of findings. Employee testing requires more careful analysis and judgement calls from the part of the researcher.

Complement findings with insights from real users

• Always follow up with user tests with real users. Internal user tests with colleagues are a good first step to refine designs and research guidelines. Making small tweaks after user tests with colleagues leads to a more effective and productive second iteration with real users.

I hope reading about some of the positive impacts we have experienced running user tests with colleagues helps reduce a flat-out aversion to internal user tests with employees. Testing with colleagues is not meant to replace testing with customers, but there are times in which testing internally can provide useful insights, as well as positive consequences such as increased transparency and trust.

Do you conduct user tests with employees? How have your experiences been? In which situations have you found it particularly useful?