Usability testing
Usability testing (or user testing) involves measuring the ease with which users can complete common tasks on your website. The results of the analysis are a huge eye-opener and their implementation often leads to:
Increased sales and task completion and a high rate of return site visitors A greatly improved understanding of your customers' needs A significant reduction in call centre enquiries A much more user-focused in-house development team Usability testing is truly an eye-opening experience. You'll have the opportunity to witness your site visitors struggle to complete what you perceive to be the easiest and most intuitive of tasks.
Deliverables from the usability testing
Written report & presentation of usability testing findings
The usability testing report is very much recommendation-led and highlights the problem areas of the site from the usability testing. Clear recommendations are provided for how to remedy each issue. Recommendations are defined as high, medium and low priority, as appropriate.
The usability testing report is written in layman's terms with all relevant issues explained thoroughly and appropriate screenshots included.
Webcredible will also make a short presentation of the main findings from the usability testing. We'll shows video clips during the presentation to highlight some of the main points. We'll then run an implementation workshop whereby we all brainstorm the best possible way to implement our recommendations.
DVDs of usability testing
We can provide you with a DVD featuring all of the usability testing videos, as well as a DVD showing some of the main highlights from the usability testing. The highlights DVD is a great way to quickly show the usability testing to colleagues who were unable to watch the testing live.
Our usability testing facilities
The usability testing takes place in our purpose-built usability lab, complete with microphones, cameras and eye tracking. Using state-of-the-art software we're able to capture what the user is doing on-screen, together with audio commentary and a video of the user carrying out the test. There's a viewing room next door with a large one-way mirror between the two rooms so you can see the testing live, in real time.
Additionally, a 45 inch plasma TV is mounted in the corner of the viewing room. A screenshot of the participant's computer screen along with a picture of their face is displayed in real-time on the TV:
We strongly advise you watch the usability testing live - it's truly an eye-opening experience. The viewing room is an airy room with wireless Internet access, a large table and a number of comfortable chairs. You also get access to our break out area, where you can relax on our comfy sofa between sessions and during lunchtime. Refreshments are provided throughout your time here.
During the usability testing
During the usability testing participants are instructed to think aloud at all times. If they ever fall silent for more than a few seconds they're encouraged to keep talking. Our usability testers are expertly trained to ask questions and elicit thoughts in a non-biased and neutral manner.
Participants are assigned specific tasks to carry out and their ability to complete these tasks is monitored. Once the testing is complete we'll conduct a brief interview with each person. We'll typically conduct the usability testing with 6-12 participants per user group.
Before the usability testing
Before the usability testing is carried out we work closely with you to identify your precise goals for both the website and the usability testing. Together, we identify the different user groups using your site and source a number of people representative of your target audiences.
We then design and prepare a list of tasks for participants to complete so they're going through realistic journeys on your site. As standard, we run a pilot testing session before launching into the full usability testing. User-centered design (UCD) - 6 methods User-centered design (UCD) is a project approach that puts the intended users of a site at the centre of its design and development. It does this by talking directly to the user at key points in the project to make sure the site will deliver upon their requirements.
The stages are carried out in an iterative fashion, with the cycle being repeated until the project's usability objectives have been attained. This makes it critical that the participants in these methods accurately reflect the profile of your actual users.
ISO 13407 outlines four essential activities in a user-centered design project:
Requirements gathering - Understanding and specifying the context of use Requirements specification - Specifying the user and organisational requirements Design - Producing designs and prototypes Evaluation - Carrying out user-based assessment of the site
The following is a typical top-level characterisation of the most popular user-centered design methods:
Method Cost Output Sample size When to use
Focus groups Low Non-statistical Low Requirements gathering
Usability testing High Statistical & non- Low Design & evaluation statistical
Card Sorting High Statistical High Design
Participatory Low Non-statistical Low Design design
Questionnaires Low Statistical High Requirements gathering & evaluation
Interviews High Non-statistical Low Requirements gathering & evaluation
Focus groups
What are they?
A focus group involves encouraging an invited group of intended/actual users of a site (i.e. participants) to share their thoughts, feelings, attitudes and ideas on a certain subject. Organising focus groups within an organisation can also be very useful in getting buy-in to a project from within that company.
When to use
Focus groups are most often used as an input to design. They generally produce non-statistical data and are a good means of getting information about a domain (e.g. what peoples' tasks involve).
Issues
It's necessary to have an experienced moderator and analyst for a focus group to be effective.
Usability testing
What is it?
Usability testing sessions evaluate a site by collecting data from people as they use it. A person is invited to attend a session in which they'll be asked to perform a series of tasks while a moderator takes note of any difficulties they encounter.
Users can be asked to follow the think-aloud protocol which asks them to verbalise what they're doing and why they're doing it.
You can also time users to see how long it takes them to complete tasks, which is a good measure of efficiency (although you should bear in mind that using the 'think aloud' protocol will slow users down considerably).
Two specialists' time is normally required per session - one to moderate, one to note problems.
When to use
Usability testing can be used as an input to design or at the end of a project. It represents an excellent way finding out what the most likely usability problems with a site are likely to be.
Usability testing can be used generate non-statistical or statistical data.
Issues
Usability testing requires some form of design to be available to test - even if it's only on paper. Testing works best if it focuses either on gathering non-statistical feedback on a design through 'talk aloud' or statistical measures.
Card sorting What is it?
Card sorting is a method for suggesting intuitive structures/categories. A participant is presented with an unsorted pack of index cards. Each card has a statement written on it that relates to a page of the site.
The participant is asked to sort these cards into groups and then to name these groups. The results of multiple individual sorts are then combined and analysed statistically.
When to use
Card sorting is usually used as an input to design. It's an excellent way of suggesting good categories for a site's content and deriving its information architecture.
Card sorting can be used generate statistical data.
Issues
Providing participants with a trial run on some easy cards (e.g. sports, animals, etc.) can reassure about what they are expected to do and result in a more productive session.
Participatory design
What is it?
Participatory design does not just ask users opinions on design issues, but actively involves them in the design and decision-making processes.
When to use
Participatory design is usually used within a mini-project to generate prototypes that feed into an overall project's design process.
An example would be a participatory design workshop in which developers, designers and users work together to design an initial prototype. This initial prototype would then feed into a more traditional design process.
Projects which only utilise participatory design are very rare.
Issues
Participatory design sessions can be very fluid and require an experienced moderator with thorough knowledge of the domain to guide them.
Questionnaires What are they?
Questionnaires are a means of asking users for their responses to a pre-defined set of questions and are a good way of generating statistical data.
When to use
Questionnaires are usually employed when a design team:
Can only gain remote access to users of a site Is seeking a larger sample size than can be realistically achieved through direct contact
It is for this reason that questionnaires are usually administered through post or electronic means.
Issues
Questionnaires allow statistical analysis of results, which can increase a study's credibility through its scientific appearance. This makes it all the more important that the questionnaire is well-designed and asks non-biased questions.
Interviews
What are they?
An interview usually involves one interviewer speaking to one participant at a time.
The advantages of an interview are that a participant's unique point of view can be explored in detail. It is also the case that any misunderstandings between the interviewer and the participant are likely to be quickly identified and addressed.
The output of an interview is almost exclusively non-statistical - it's critical that reports of interviews are carefully analysed by experienced practitioners.
When to use
Interviews are usually employed early in the design process in order to gain a more detailed understanding of a domain/area of activity or specific requirements.
Issues
Interviewing places a high premium on the experience and skill of the interviewer and analyst.
Conclusion This has been an introduction to the major user-centered design methods. It's vital to remember that although each can be extremely valuable, using them in the right way, for the right reasons and at the right time is critical.
Exactly which method to use, and when and how to use it will differ from project to project. Split A/B testing
Following on from any website usability study a number of usability problems are usually found. There can often be debate within any organisation as to the best solution for each problem, with no one really knowing the optimal solution. Rather than letting the person that shouts the loudest get his or her own way, a better solution can be to test 2 solutions in a live environment. Whichever performs the best is clearly the superior solution. Welcome to split A/B testing!
Buridan's ass and A/B testing
Did you hear the story of the donkey stood in the middle of 2 equally appealing stacks of hay? He spent so long trying to decide which one to start eating first that the poor animal starved to death without having moved a single inch. This paradox, first discussed by Aristotle is known as 'Buridan's ass' and is an often discussed psychological phenomenon.
If you're managing a website you might face similar situations when you need to decide which of 2 different designs to opt for. Nowadays, you need to continuously improve and evolve your site by making small frequent adjustments. But how do you know which change will have the highest impact on the customer experience?
Split A/B testing is a way of finding out which changes help your users' performance. It provides a controlled method of measuring the effectiveness (or not) of alterations to your site. It's often used for small tweaks (e.g. 'Is this style heading clearer than the original?') but can also be used to test bigger wholesale changes (e.g. 'Is this new 1-click checkout process better?'). In essence, it involves running 2 different versions side by side to see which is more effective.
A typical A/B scenario
You feel your users might not be finding the 'Proceed to checkout' button hidden below the fold of the page and think this might be causing people to leave your site before making a purchase. You've created a new design which you feel is more effective but you'd like to know for sure.
To A/B test your new page, you serve the regular page to say 90% of your visitors as usual, but a randomly selected 10% would be shown your new design. Then you sit back, pour yourself an ice cold gin and tonic, wait and watch your web statistics. If your new page has a positive effect as you suspect, then you should see an improvement in the 10% group, as measured by conversion rates. Proof that you should publish the more successful page to all visitors.
Advantages of split A/B testing
There are a number of benefits to A/B testing:
Low risk approach Cheaper than other methods such as focus groups Provides proof Invisible to most of your users Great way to do 'test run' new designs before full roll-out (to avoid negative surprises on the launch day) Can solve internal disputes
Disadvantages of split A/B testing
A/B testing isn't always suitable. Some of its disadvantages include:
New designs might have to 'wear in' before you can measure their real performance (visitors' initial response might be negative because they're used to the old solution) You can only compare 2 versions with a single factor that differentiates both designs (for more factors / variations you need to deploy multivariate testing which is more difficult to analyse) It takes technical know-how to set up and analyse the results You're testing in a live environment so external factors might have an impact on the outcome If your site is really in bad shape, you'll still need to do a proper overhaul
In a nutshell
A/B testing is powerful stuff and a useful method for a quick comparison for 2 different designs. However, bear in mind that it's no substitute for getting proper feedback from your users. Only this will give you the whole picture straight from the ass's mouth. Expert usability review vs. usability testing
One question we're often asked is which method is best: usability testing or expert usability reviews? Well, if they were sports cars, expert usability reviews might be a Porsche (pretty decent car and better than no car at all), but usability testing would be in a different league, namely Formula 1.
So, what's the difference?
An expert usability review is when a usability specialist inspects a website to identify potential usability problems Usability testing involves getting people from the target audience to evaluate your site whilst performing tasks
They're quite similar in many ways in that both:
Find and prioritise usability problems Evaluate designs in the context of tasks
Do they find the same problems?
The answer depends on how precisely you measure this, on balance no, not reliably. Generally speaking, expert usability reviews tend to find high level breaches of design rules and consistency. Usability testing is better at finding issues related to special domain knowledge and task flows.
If you were to carry out an expert usability review before conducting usability testing, and compare the two sets of findings, you can see how the issues you found compared to their real experiences. Did you find all the problems? Which ones did you miss? Which ones that you thought might be issues turned out not to be? This will give you a good feel for how expert usability reviews hold up against usability testing.
Often, expert reviews will:
Miss usability issues that arise during usability testing Find some issues that usability testing didn't Report false alarms (i.e. not real issues)
And that's the problem. No matter how good an expert you are if you just rely on an expert usability review you'll find different sorts of issues and overlook some potentially serious ones.
Consider the case of a health website aimed specifically at new mums. The expert usability review found plenty of great issues, but might have missed the main killer one that caused participants to pass comments like “That's just so insensitive!” and “I wouldn't use this site anymore”. And all because of a seemingly benign bit of banner advertising for a local gym placed next to an article about breast feeding. A reviewer would have difficulty predicting the indignant reaction of new mothers who felt the site was rubbing their nose in it. After all there would be little chance of them going anywhere near a gym with a newborn. How much overlap is there between expert usability reviews & usability testing?
Naturally, there is some overlap in the problems found in both expert usability reviews and usability testing. Research conducted by Fu, Salvendy and Turley in 2002 estimated this to be around 41%. Not as high as you might expect. Interestingly, expert usability reviews tend to find more problems compared to usability testing. But it's quality over quantity.
Expert usability reviews are good...
Expert usability reviews are more common than usability testing simply because they're cheaper and quicker to do. Typically, it'll only take a few days to inspect a site and write a report and because you don't need to hire labs nor recruit users, costs are significantly cheaper. Because of this, they're often preferred to usability testing, especially when time and budget are squeezed.
There are however some pitfalls. For a start, a lot depends on the reviewer's level of experience. No two usability experts will find exactly the same issues so the method is also open to practitioner variation. The biggest problem however is simply that they sometimes miss the real problems that cause users to fail tasks. This can be especially true when the target audience has a particular skill set. For example, an accountant may be better placed to diagnose certain types of issues related to online accountancy tasks than a non-accountant reviewer.
...But usability testing is da Daddy!
Usability testing takes more time to plan and organise and is more expensive too - recruiting and incentivising people from a target audience can be costly. But it's definitely worth the effort! The results give a truer picture of the real problems people encounter because they're derived from real users in the first place.
It's inconceivable how sites go live without checking to see that they're actually usable. It's like building a car but not test driving it. "Yes, we've built this great website. Umm, well no, we've not checked to see if people can use it yet. But don't worry because all the links work and the pages download quickly."
Another advantage of usability testing is that there's less conjecture and feedback comes straight from the horse's mouth. Videos of user sessions in particular, can provide a powerful persuasion tool for reluctant stakeholders. It's hard to argue with recordings of people in tears of frustration.
Use real users and accept no substitute!
Both expert usability reviews and usability testing have their time and place. In practice, people often use expert usability reviews early on to straighten up their design in preparation for usability testing. (Indeed this is critical for accessibility testing, since it's a waste of time to recruit and test users who can't even get into your site in the first place.)
Whilst an expert usability review is better than nothing, it's still vital to put your website in front of users as early as possible too, rather than leaving it too late. Key elements such as site structure and navigation should be assessed with users at the earliest opportunity. Otherwise you risk having your website completely overhauled at a late stage.
Ultimately, the key is to appreciate that expert usability reviews and usability testing are different beasts. The most effective approach is to try and integrate both techniques. Remember though, you must test your site with real users - accept no substitute!
Have us conduct usability testing for you!
Find out whether site visitors can actually use your website by having us run usability testing for you.