Usability testing - now and the future

16th October 2007

SPiN logo (Society of Public Information Networks)

All councils and public sector organisations have a clear remit to make their information available and accessible to their citizens, and increasingly they are applying usability testing to gather empirical evidence of the effectiveness of their sites. 

Chris Rourke, Managing Director of User Vision goes on to say whether this is done by internal teams or external, independent organisations and consultants, usability testing is now part of a process applied by many site managers.  Indeed usability is considered by  SOCITM in their annual UK council site reviews, and is a critical part to citizen fining services and transacting online.

Key Aspects

At its basic level, usability testing involves asking your target users to perform key tasks on your site and observing carefully, and probing or discussing with them as appropriate.  As opposed to focus groups, usability tests typically involve one person at a time.  One of the perennial questions in usability research is ‘How many subjects are needed for usability resting?'  Many usability experts find that testing with even as few as 5 or 6 subjects can provide reliable, accurate insight upon which recommendations can be made.  Whatever the number of subjects, there are key aspects that will make the difference between a good and a poor usability test.

  • Accurate recruitment if subjects - involving people from your key demographic or target audience.
  • Methodical test planning - creating clear, non leading task scenarios that probe the important parts of the site, give the user a goal, but do not tell them how to achieve it, so that correct navigation and site use us measured.
  • Effective ‘think aloud' moderation by the consultant or staff leading to the test which avoids leading questions and probes where appropriately.
  • Clear, balance reporting of issues, pointing out the positive as well as negative aspects to the design.

Viewing standard usability testing through a one-way mirror

This method is fairly well established and there are several books and websites providing guidance on standard usability testing procedure.  But is the current method the state of the art or is there more that can be learned through other methods? Some of the more advanced methods that may be considered are:

Eye tracking - what are you looking at

One of the exciting modern methods that can be applied is eye tracking which, as the name implies, involves studying the eye movements of the usability test subject an applying the lesions learned to optimise the design of the web pages and the entire site.  The method is applied as part of usability testing, but requires specialist equipment to monitor and analyse the eye movements.  The user's eye movements can be studied in real time, to see the sequence of items that an individual looked at, or they can be examined across several people at once, to learn, what in general, people looked at on a given page.  Some of the visual outputs from eye tracking include gaze trails (showing the order in which the person's eyes moved across the screen) and heatmaps (showing the cumulative intensity with which people viewed different parts of the screen).

Although visually compelling, the outputs from eye tracking only tell half the story at most.  The images such as the ones shown above need to be interpreted carefully with consideration of the user's verbal description of their thoughts, actions and decisions during the task performance.

Although eye tracking can be applied to any web page, it is often most effective when applied to key pages where the design of the calls to action and navigation structure will make a clear difference to user's actions.  Another good implementation is comparative usability testing between 2 or more design options for a single page.  A carefully designed and conducted test can show which one is working best for providing orientation to the user performing critical tasks.

Screenshot showing heatspots on the elibrary website

Heatmap of page showing how several people tended to view a page 

Remote moderated usability testing

Traditional usability testing involved the test subject sitting side by side with the test moderator, and typically some incentive is required to attract them to the testing location, but what do you do if you have a widely distributed client base? Often it is not financially feasible to travel from place to place to test individual test subjects due to the time lost in travel and test set up.

Thanks to developments in modern web-based screen sharing and conferencing systems, remote usability testing, where the test subject and test moderator are separated by hundreds or even thousands of miles, is now a realistic option.  Programs such as Webex, Go To Meeting and others allow the test moderator to see the screen the user is viewing, as well as the cursor movements, and a straightforward audio connection through phone or web conferencing carries the users verbal descriptions of their actions and opinions.

There is a downside to remote testing, since that rapport built up over the phone may not be as strong as face to face discussion, and of course there is no opportunity to interpret any body language, which can be an interesting metric in traditional usability tests.  Inevitably there are more logistical challenges and with the greater reliance on technology there is a risk of unforeseen problems, but for public organisations seeking to capture the opinions and behaviour of their far flung citizens, remote usability testing is a very effective method to keep in mind. 

Remote volume usability testing

One of the main differences between usability testing and market research based methods is that the number of subjects involved in the research tends to be much less.  As we have seen, testing with a small number can provide useful results, but if you want to see what lots of people are doing perhaps on specific tasks and capturing their views on their experience of using the site?

Remove usability testing can be performed in an unmoderated format to capture information on the experience of targeted users performing specific tasks.  Unlike web analytics, which is simply looking at what users did on your site user can be invited to perform specific tasks (e.g. find if it is possible to reserve a book at the Council library through the website) and their performance in the task can be recorded both an a personal basis and also collectively.   

The large numbers of people performing the task will also analysis of all aspects of the task, providing data similar to standard web analytics, but it also allows the presentation of questions during and after the task.  This allows important metrics like the success rate to be tracked, as well as qualitative information about what the user thought of the experience.

Screenshot of the Bank of Scotland website showing the gaze plot of a test participant

Gaze trail of a site showing the order in which a user viewed a page. 

A/B and multivariate testing

Finally, a method related to usability testing which can be very effective for refining a site design is A/B and multivariate testing.  This method involves offering two or more variations live of a web page, and site visitors clicking through to the page are randomly split regarding which version of the page they see.  By carefully tracking the performance of the user and what they do on that page, evidence quickly builds as to which version is most effective. 

For example if you have a page that outlines a council's range of online services, A/B testing would allow you compare two different versions of the page to see which one is working better to lead users through to an important conversion, such as performing transactions online.  The differences could be one of many factors such as the clarity of the call to action, the labelling of links, or the graphic design.   The user never knows that they are on an experimental page, and they never are exposed to the other versions of the page, but with enough traffic through these selected pages it quickly becomes apparent which page design or content is most effective.

If you already have a fair degree of confidence of the usability of your site, you may be in a position to apply some of these advanced usability testing methods to refine the usability of your current and future site.

Chris Rourke

What can you do next?

Want this article on your website?

If you liked this article, feel free to republish it on your own website. All that we ask is that you include the citation below, including links, at the end of the article.

This article was written by Chris Rourke. Chris is the Managing Director of User Vision, a usability and accessibility consultancy that helps clients gain a competitive advantage through improved ease of use.

Research indicates that £1 spent on advertising yielded £5 in total revenue, while £1 spent on customer experience improvements produced more than £60.

Creative Good.