A formal or informal expert usability review is a common starting point for most of the projects we undertake in-house. Oddly enough it’s also one of the only methods in a user centred design process that doesn’t directly involve people who will use the product or service.
That said, we still find it a useful way to capture our “initial thoughts” when we use a website for the first time. This is particularly useful for longer term engagements when you’re likely to be collaborating directly with an in-house team.
What is an expert usability review?
An expert usability review is a method used to assess how usable a website is from an end user(s) perspective, in the absence of usability testing.
The focus of an expert review is to identify issues people are likely to encounter when they use a digital product or service.
In short, an expert usability review should:
- Provide objective analysis
- Evaluate designs in the context of the tasks (prioritised end-user segments) wish to complete
- Find and prioritise usability problems.
Even though an expert review cannot offer the depth or confidence that observing real end-users of a product or service can. It is still a useful method for rapidly (and cost-effectively) sharing independent advice throughout a project’s lifecycle. Hence, its popularity.
How to conduct a useful expert usability review?
Best practices are just the start
An expert review should be informed by heuristics and human computer interaction. However, we believe it should go beyond this too.
Take a user centred approach
A good expert review shouldn’t just rely on best practices it should also be informed by who the specific users of the product or service are/will be. These should be segmented by intent or behaviour and where possible you should try to define their likely:
- Customer (or user) journey
- Context(s) of use
- Motivation and purpose for using the product or service (including a granular understanding of the tasks they are looking to complete).
Understand the business
In addition to this the consultant must have a sound understanding of the:
- Goals of the product or service
- The company’s business model
- Any relevant sales and marketing objectives
- Any relevant (offline or online) customer experience issues.
Involve multiple consultants
At Border Crossing UX we always use at least two consultants to conduct and document an expert review. Why? Given the method, it is absolutely critical that you get more than one consultant to conduct a review. No matter how good the consultant it’s impossible for one person to find all of the different issues a user may encounter whilst using a website.
Complete the following steps
- Understand (and segment) who the users of the product or service
- Use the website as if they were a prospect or customer
- Leverage their knowledge and experience of testing websites with users
- Identify issues and recommend changes to improve usability.
If budget allows, we would always recommend that you replace or supplement an expert review with first-hand user research and usability testing.
Our process for conducting an expert usability review
Oddly enough the best thing you can do when starting an expert review is to NOT look or use the website you’ve been tasked with reviewing.
This is critical as you only have one chance to see it, and truly think like, a first-time user. Instead we’d recommend that you:
- Identify key (business) goals
- Define key user segments
- Develop a persona per segment (motivation, context of use, top tasks)
- Task analysis (identify the information and steps required to achieve each task)
- Conduct the critical tasks
- Document findings
- Analyse and group findings
- Present the findings.
How accurate are expert usability reviews?
There has been extensive research conducted on the accuracy and veracity of expert usability reviews and usability testing.
Studies indicate that there is up to a 41% overlap in the problems highlighted when conducting an expert review and usability testing, (Fu, Salvendy and Turley).
Expert usability reviews tend to find more problems compared to usability testing. But note that an expert review may still miss serious usability issues that are more likely to be found if testing with users.
What are the benefits of an expert usability review?
- Fast (turnaround times should be measured in days and not weeks)
- Low(er) cost
- Independent feedback
- Identify minor issues (that may be missed in a usability test)
- Greater scope (an expert review can evaluate more parts of a user interface than a usability test which tend to be more focused) which in turn can lead to more issues being identified
- Timing (you can collect feedback at pretty much any stage of the product or services lifecycle, e.g. design, development, pre or post launch)
- Useful starting point that should inform what needs to be examined further.
What are the drawbacks of an expert usability review?
- Usability issues will be missed (it is possible to miss serious underlying issues)
- False alarms can be raised (not real issues)
- Frequency (it is not possible for an expert review to identify how often a user is likely to encounter an issue highlighted)
- Consultants are not likely to be authentic and representative (real-world users’ mental models, experiences and expectations will be different to that of the consultant(s) conducting the review)
- Quality is dependent on the knowledge, experience and skills of the reviewer
- Number of consultants required (multiple experts should be used as different reviewers will do things differently which leads to more issues being identified)
- User behaviour (watching real users use a website for the first-time often highlights one or two surprising issues that are nigh on impossible to anticipate).
When to consider an expert usability review?
One of the biggest benefits of an expert review is that you can do it any time.
Developing a new product or service? Commission one to inform the development of your wireframes or front-end designs.
Trying to improve an existing product or service? Commission an expert review to augment usability tests or identify what to test first.
When to consider usability testing?
As we’ve mentioned an expert usability review is great for finding high-level issues. But usability testing is certainly our preferred option if there is:
- The time, budget and internal buy-in to do this
- Limited internal buy-in (seeing firsthand evidence of the issues end-users are struggling with is a key tool in cutting through internal politics)
- Limited information to hand with regards to the actual/potential users of the product or service.
This is because an expert usability review can never compete with insights generated from (remote or on-site) usability tests with real end-users of the product or service. This is particularly true if a product or service is tightly focussed on the needs of domain experts and their specific task flows. As in these instances it’s nigh on impossible for a consultant to act as a representative user.
Should I combine an expert review with usability testing?
Yes, if the budget and time is available. This is because, research by Law and Hvannberg (2002) has shown that usability testing and expert reviews tend to highlight quite different issues.
Expert reviews tend to highlight more problems with labelling, design and consistency. Whereas usability testing tends to highlight issues with task flows (as in the order a user would expect to complete a series of steps). What’s more further research by Rasmussen indicates that expert reviews identified significantly more skill-based and rule-based issues. Usability testing identified more issues at the knowledge-based level of behaviour.
This should be altogether surprising. As knowledge-based challenges arise when users are learning or modifying their mental model while completing a task.
The conclusion that Fu, Salvendy and Turley (2002) came up with is that the most effective approach is to integrate both expert reviews and usability testing at different times in the design process. Their analysis indicates that an expert review is best applied first or early in the design process. Once these issues have been resolved, usability testing can be used to focus on the higher-level issues that may be missed in the course of an expert review.
Their research also indicated that if:
- The scope of changes are minimal, e.g. layout changes, an expert review is likely to suffice.
- The scope of changes are extensive and/or complex, e.g. substantial information architecture changes, then usability
- testing with representative users becomes increasingly important.
How should I evaluate my website?
Well it depends. Here are a few parameters that should help guide your decision on the best way to evaluate your product or service.
- Expert usability reviews and (remote or on-site) usability testing are likely to highlight different (types of) issues
- Combining techniques is likely to increase the number of issues found
- Expert usability reviews only work well when:
a) Specialists do them
b) Limited domain expertise is required to complete the tasks to be assessed
c) You already have internal buy-in to fix the issues highlighted by the report.
Want an independent evaluation of your website?
Could an expert review or UX audit help you benchmark your performance and help you focus on what matters most?
Get in touch to discuss your objectives and how we can help you achieve them.
