Photo: K2_UX
Thankfully the majority of businesses have come to appreciate that user testing is an invaluable tool in digital product design. Done right, this additional investment in product development pays dividends.
But, businesses embarking on usability testing for the first time may find that unlike its established and well-regulated older brother, market research, there is little documented best practice or recognition for it as a specialist discipline.
Like all types of research, unfortunately it’s easy to get it wrong and I have seen many instances where simple mistakes have led to usability testing being unproductive, and even harmful to the product.
Whether you’re commissioning, conducting or observing usability testing, make sure you recognise these common mistakes.
1. Believing that user testing is free
User testing, formal or informal, is not free, not quick and not particularly easy. In a small organisation without dedicated researchers, in-sourcing isn’t necessarily cost effective, or even effective. There’s an expectation that user testing should fall into a designer’s remit but this doesn’t make sense when considering the diverse tasks involved:
- Recruiting participants – Defining the participant profile, contacting and screening users, booking them, reminding them
- Logistics – Scheduling people, booking rooms, booking or buying equipment
- Planning – Designing tasks, writing a discussion guide, preparing prototypes and other testing materials
- Running the sessions – Greeting participants, facilitating the sessions, taking notes
- Evaluating – Reviewing notes or videos, collating feedback, making recommendations, presenting
It’s not uncommon for even a small study to require 10+ staff-days (excluding observation time) – quite an ask on top of an existing busy workload – and may take six weeks from start to finish. [There are other reasons why designers may not be best placed to conduct user testing, see #13].
Takeaways
- It’s worth outsourcing at least some of the tasks to specialists, especially recruitment
- Recognising the investment required, read on and make sure it’s a success!
2. Not having clear objectives
So we’ve established the user testing can be a considerable investment in time and resources.
Just like any other type of research, in order to get value from user testing you need to have clear objectives, and the easiest way to do this is to define a set of research questions. It sounds obvious, but all too often this step is forgotten. Face-to-face research can help you answer the ‘why?’ where analytics and other metrics can only give you the ‘what?’. Research questions should be specific, and might include things like ‘Why is exit rate high payment screen?’, ‘Do users understand these icons?’, ‘What do users expect to see on their account dashboard?’. [Be wary of ‘Do users like…?’ questions, see #11]
Takeaways
- Write a list of research questions – the things you’re hoping to discover by conducting the research.
- With this starting point you can outline the interview questions, tasks, materials and participant profiles that will enable you to answer these research questions.
3. Seeking validation
Following on from #2, the purpose of any usability study should be to better understand users and identify ways that products can be improved. If, when your design team is asked why they’re conducting usability testing, they reply with something like “to validate our ideas”, alarm bells should ring. This is research as a box-ticking exercise, not a learning exercise. It’s a failure to acknowledge that design is based on assumption, that there’s always room for improvement. It’s also a sign that there is no innovation – innovation requires risk and risk requires thoughtful evaluation.
Takeaways
- Present several alternatives to see which perform best.
- Explore the existing product or competitors alongside the new.
- Approach user testing with a willingness to throw everything away and start again afterwards
4. Recruiting the wrong participants
Many companies, when asked who the target audience for their product is, reply “everyone”. This is almost never the case, and it’s a particularly dangerous assumption when recruiting participants for user testing. Testing with people who would never use the product will lead to negatively skewed results.
If the conclusions of your research don’t match the real-world user behaviour there’s a good chance this has happened; for example if your usability tests report that only 2/10 users complete a checkout but your real-world conversion rates are high.
A real-world example: While conducting a study into a UGC product, I interviewed someone who had never and would never share a photo on social media. She had been recruited based on the audience profile supplied by the client but predictably, couldn’t understand the point of the product. Another common example: testing an Android app with iPhone users, who suggest ways that it should be more like an iPhone app.
Takeaways
- Recruit users who already exhibit the behaviours you’re looking for, not just the target socio-economic profile. For example, if you’re testing an ecommerce product, recruit people who already buy online and have an interest in the type of product on offer.
- Always conduct a brief contextual interview – it can help explain anomalous results.
- If you’ve had very negative feedback from user testing, don’t panic! Review the contextual interviews to consider whether the participants represented real-life users. Learn from it and refine the screening criteria for next time.
- Guerilla testing can be equally flawed for the same reason – be wary of results unless the subject is very generic.
5. Recruiting the wrong users
Yes, this is subtly different from the previous point. You can still get skewed results when all your participants are existing users, if you’ve managed to find an unrepresentative subset.
For example, recruiting participants through your Facebook page, you can end up talking to only the most engaged subset, or conversely, people who have never engaged with your product, they follow your page for another reason such as the content your post or that giveaway your marketing department ran. Either way they are atypical.
In a real-life example, a client of mine ran an email survey with existing customers to determine business priorities. The survey gathered over 2000 responses and was considered a huge success, but using the raw results to inform strategy would have been a huge mistake. On inspection the respondents were not a representative sample and were heavily weighted towards older customers. This is easily explained considering that the survey was sent only to customers that had opted in to receive marketing emails, and that the lengthy survey would not have been completed by anyone without time on their hands. While the number of results was in theory enough to draw statistically valid conclusions, it actually only represented 2% of the customer base.
Takeaways
- Recruiting through your existing channels can save money, but it’s essential to conduct the same behavioural and demographic screening or it’s highly likely you’ll have an unrepresentative sample.
- Check social media demographics against the demographics of the wider users base before using social channels to recruit.
- Use statistical weighting to balance survey data towards a representative demographic.
6. Testing users, not usability
User testing can be nerve wracking for those taking part, especially if they’re not tech savvy, or the environment is intimidating – for example if there are too many people in the room or visible recording equipment. Bizarrely, the more uncomfortable they feel, the more positive they will be about what they’re doing because they feel that the problem must be them, not the software. If a user asks “How did I do?” you can be pretty sure this is happening, and you’re getting a rose-tinted view if the real life user perspective.
Takeaways
- Hiring a dedicated lab with a separate observation room is a great idea, but if this is beyond your budget, why not simply use a webcam to set up video conferencing to another room?
- Don’t allow anyone into the room that doesn’t need to be there. If someone other than the moderator needs to be present, make sure they introduce themselves and the participant understands their role.
- Kick off sessions with a contextual interview. Apart from giving useful insights, this can help participants to relax and feel valued.
7. Making it too hypothetical
Asking a user to imagine that want to do something they’ve never done with a system they’ve never used is unlikely to yield useful feedback. It’s important to try to get participants to buy-in to the task by providing context and making it as realistic as possible.
In a real-world example for online print company MOO, we tested a design interface for business cards and offered to print what participants designed for free, asking them to bring their existing business cards and any company logos. In doing this we identified real-life issues that we would otherwise have missed (e.g. users wanting to add more details than the designs allowed, struggles with image formats for logos), observing a genuine level of frustration when they were not able to achieve what they wanted. We also noted that users forgot they were getting the product for free, and behaved as if they were actually paying, choosing finishes and options based on their budget.
Takeaways
- Make the tasks as realistic as possible, or real if you can! Giving away a few goods will pay for itself in insights.
- If using prototypes or mocks, make sure they show real content (don’t waste time having to explain what Lorem Ipsum is).
- Set the context by creating end-to-end journeys, for example by starting the task from a google search.
8. Alienating participants
Never underestimate the strength of personal taste. Asking users to perform a task with sample content that doesn’t appeal (or actively hate) is a sure way to put them off.
For example, when testing a video-on-demand app for a broadcaster, we noted that participants responded very negatively to the feature we were testing if they disliked like the sample TV content we’d prepared, regardless of whether they usability issues or not. For future sessions, we asked participants about their TV genre preferences as part of the screening process, enabling us to prepare appropriate sample content in advance. This resulted in feedback being focused on the interface, not the content.
Takeaways
- If possible, find out what users buy/watch/do in advance, as part of the screening process and prepare several sets of content as appropriate.
- Use the contextual interview to discover users’ preferences, then relate the task to the user. E.g. “You mentioned you like nature documentaries, how would you go about finding something to watch using this app?”
9. Buggy software
While building fully-functional software for user testing is often not the right use of resources, if your prototype contains more bugs than working features, your main take-aways from user testing will be that there are bugs and that bugs are terrible for usability. [Aside: This is actually a valuable lesson in itself, and one worth demonstrating first hand if you’re struggling to make a case for bug fixes.]
If a participant encounters multiple errors, no matter how much the facilitator tries to reassure them “don’t worry, it’s not you, it’s the prototype”, the participant will start to alter their behaviour to avoid triggering an error and any chance of realistic feedback goes out the window.
Takeaways
- If testing with a technical prototype, ensure it’s working well enough that it’s not a game of “hunt the working button”.
- If you can’t do that, clickable mocks or even paper prototypes are likely to give more useful insights.
10. Introducing technology barriers
If you’re testing a technical prototype, give the participant a platform they’re familiar with on which to use it; don’t let a Windows user struggle with a Mac mouse, don’t ask an iPhone user to browse on an Android device. You will find it difficult to distinguish problems relating the unfamiliar platform from usability issues with the feature you’re testing.
If testing browser-based software, I like to provide both a Mac and a Windows machine and ask participants which they prefer, and let them use their own phone or tablet.
Takeaways
- Screening provides an opportunity to discover in advance what technology the participants are familiar with, so make sure what you provide is appropriate.
- Find out what web browser the user prefers as this can also impact their ability to navigate the prototype.
- Choose your words carefully and avoid jargon – ‘What do you use to access the internet?’ is better than ‘What web browser do you use?’ to a novice user.
- While you’re at it, make sure they’ve got a comfortable chair, sensible-height screen and a mouse (don’t let participants struggle with a track-pad).
11. Listening to opinions over actions
What participants say and what they do are often very different. Be wary of asking users what they think about something or putting too much weight on strong opinions. A user may have a very vocal reaction to something (e.g. a colour) that’s inconsequential to their usage of the product.
There is a design urban myth that (whether true or not) perfectly illustrates this phenomenon: During a focus group for the classic Sony Boombox, participants were asked whether they preferred the yellow or black version. All stated that they preferred the yellow. At the end of the session, participants were offered the choice of one to take home. While they may have loved the idea of the edgy yellow boombox, all took home the black.
There may be times when user preferences are at odds with design objectives. A real world example: When designing part of an early mobile web interface we tested a number of variations of a carousel. Participants overwhelmingly said they preferred a version with manual controls, but not one of them actually explored the carousel content unprompted, and would only have seen the first item in real life. An autoplay carousel was much more successful because users saw (and commented on) content beyond the first item.
Takeaways
- Actions speak louder than words. Design tasks that tell you about their behaviour, not their opinions.
- Be careful to give learnings from quieter participants as much weight as those from vocal participants.
12. Expecting users to be designers
It can be tempting to ask participants how they would design an interface, and it can be compelling to hear their ideas first hand, but it is very unlikely to lead to product innovation. Non-designers will typically only think of features that are within their existing frame of reference, i.e. products they’ve already used. This not only makes it hard for them to generate novel ideas, but difficult for you to identify the fundamental user need.
It’s likely that there’s a better solution to their problem that might not even exist yet, and the best people to work out what that is are skilled and experienced in doing just that: designers.
Takeaways
- Find out what users need from a product, rather than asking them to design it.
- Tasks such as a card sort can help determine user priorities, while being abstract enough not to get bogged down on design details.
13. Designers conducting their own usability testing
As discussed in #1, there’s no reason why designers can’t make great researchers but while being able to interpret research is an important skill in in user experience designer, being able to conduct research is not. Quite apart from the skillset required, there are two very good reasons why a person who has designed a feature shouldn’t conduct its user testing:
- Cognitive bias – No matter how aware the designer is of their own bias, they subconsciously or consciously design tasks, ask questions and evaluate responses in a way that validates their own view (see #3)
- If designers are facilitating, they aren’t observing
Takeaways
- Get designers to facilitate and evaluate testing for each other’s work if you’re confident they have research skills.
- If you can’t justify hiring a full time researcher, use a freelancer.
14. Leaving it too late
This is the single biggest and most common mistake that organisations make when launching new features. There is absolutely no point user testing unless you have a willingness to make changes. As discussed in #3, usability isn’t a simple box that can be ticked.
In theory, user testing fits well with Agile’s “Fail early, fail often” mantra but the reality is often different. Designers and researchers become unpopular when they recommend fundamental changes and blame is thrown around “Why didn’t you get it right the first time?”. As launch approaches, even minor UI tweaks are going to jeopardise the delivery roadmap.
Takeaways
- Schedule development time for iterations after user testing.
- Test early, test often – Regular, informal user testing is highly effective when built-into an iterative design process.
- [At risk of being controversial] If you’re that close to launch, just launch. Gather feedback from real users and analytics data once live, then iterate quickly.
Conclusions
If I’ve made user testing sound somewhat overwhelming, please don’t be put off. User testing is always a learning experience, not just about the product but the research process itself. Always evaluate the methodology as well as the feedback, and refine practices over time. And if you need help, research specialists are on hand!
