Evaluation: Inspections, Analytics & Models

Chapter 15
Evaluation: Inspections, Analytics & Models

Aims:
• Describe the key concepts associated with
inspection methods.
• Explain how to do heuristic evaluation and
walkthroughs.
• Explain the role of analytics in evaluation.
• Describe how to use Fitts’ Law – a predictive
model.
www.id-book.com 2
Inspections
• Several kinds.
• Experts use their knowledge of users & technology to

review software usability.
• Expert critiques can be formal or informal.
• Heuristic evaluation is a review guided by a set of

heuristics.
• Walkthroughs involve stepping through a pre-planned

scenario noting potential problems.
www.id-book.com 3
Heuristic evaluation
• Developed by Jacob Nielsen in the early 1990s.
• Based on heuristics distilled from an empirical
analysis of 249 usability problems.
• These heuristics have been revised for current
technology by Nielsen and others for:
– mobile devices,
– wearables,
– virtual worlds, etc.
• Design guidelines form a basis for developing

heuristics.
www.id-book.com 4
Revised version (2014) of Nielsen’s original
heuristics
• Visibility of system status.

• Match between system and real world.
• User control and freedom.
• Consistency and standards.
• Error prevention.
• Recognition rather than recall.
• Flexibility and efficiency of use.
• Aesthetic and minimalist design.
• Help users recognize, diagnose, recover from errors.
• Help and documentation.
www.id-book.com 5
No. of evaluators & problems
www.id-book.com 6
Number of evaluators
• Nielsen suggests that on average 5

evaluators identify 75-80% of usability
problems.
• Cockton and Woolrych (2001) point out
that the number of users needed to find
75-80% of usability problems depends on
the context and nature of the problems.
www.id-book.com 7
Heuristics for websites focus on key
criteria (Budd, 2007)
• Clarity
• Minimize unnecessary complexity &
cognitive load
• Provide users with context
• Promote positive & pleasurable user
experience
www.id-book.com 8
Using heuristics to evaluate to evaluate
ambient displays
www.id-book.com 9
3 stages for doing heuristic
evaluation
• Briefing session to tell experts what to do.
• Evaluation period of 1-2 hours in which:
– Each expert works separately;
– Take one pass to get a feel for the product;
– Take a second pass to focus on specific features.
• Debriefing session in which experts work

together to prioritize problems.
www.id-book.com 10
Advantages and problems
• Few ethical & practical issues to consider
because users not involved.
• Can be difficult & expensive to find experts.
• Best experts have knowledge of application
domain & users.
• Biggest problems:
– Important problems may get missed;
– Many trivial problems are often identified;
– Experts have biases.
www.id-book.com 11
Cognitive walkthroughs
• Focus on ease of learning.
• Designer presents an aspect of the
design & usage scenarios.
• Expert is told the assumptions about user
population, context of use, task details.
• One or more experts walk through the
design prototype with the scenario.
• Experts are guided by 3 questions.
www.id-book.com 12
The 3 questions
• Will the correct action be sufficiently evident
to the user?
• Will the user notice that the correct action is
available?
• Will the user associate and interpret the
response from the action correctly?
As the experts work through the scenario

they note problems.
www.id-book.com 13
Pluralistic walkthrough
• Variation on the cognitive walkthrough theme.
• Performed by a carefully managed team.

• The panel of experts begins by working separately.
• Then there is managed discussion that leads to
agreed decisions.
• The approach lends itself well to participatory design.
• Also other adaptations of basic cognitive
walkthroughs.
www.id-book.com 14
Evaluation using analytics
• A method for
evaluating user
traffic through a
system or part of a
system.
• Many examples:
Google Analytics
(chapter 7),
Visistat (shown
below), Learning
Analytics.
• Times of day &

visitor IP
addresses
www.id-book.com 15
Social action analysis
(Perer & Shneiderman, 2008)
www.id-book.com 16
Predictive models
• Provide a way of evaluating products or
designs without directly involving users.
• Less expensive than user testing.
• Usefulness limited to systems with

predictable tasks - e.g., telephone answering
systems, mobiles, cell and smart phones.
• Based on expert error-free behavior.
www.id-book.com 17
Fitts’ Law (Fitts, 1954)
• Fitts’ Law predicts that the time to point at an
object using a device is a function of the
distance from the target object & the object’s
size.
• The further away and the smaller the object,
the longer the time to locate it and point to it.
• Fitts’ Law is useful for evaluating systems for
which the time to locate an object is
important, e.g., a cell and smart phones,
a handheld and mobile devices.
www.id-book.com 18
Key points
• Inspections can be used to evaluate requirements, mockups, functional
prototypes, or systems.
• User testing & heuristic evaluation may reveal different usability problems.
• Design guidelines can be used to develop heuristics
• Walkthroughs are focused so are suitable for evaluating small parts of a

product.
• Analytics involves collecting data about users activity on a website or

product
• Fitts’ Law can be used to predict expert, error-free performance for clearly
defined tasks with limited key presses, eg. data entry and smart phone
use.
www.id-book.com 19

Evaluation: Inspections, Analytics & Models

Uploaded by

Copyright:

Available Formats

Evaluation: Inspections, Analytics & Models

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Evaluation: Inspections, Analytics & Models

Uploaded by

Copyright:

Available Formats

Chapter 15

Evaluation: Inspections, Analytics & Models

• Experts use their knowledge of users & technology to

• Expert critiques can be formal or informal.

• Heuristic evaluation is a review guided by a set of

• Walkthroughs involve stepping through a pre-planned

• Design guidelines form a basis for developing

• Visibility of system status.

• Nielsen suggests that on average 5

– Take one pass to get a feel for the product;

– Take a second pass to focus on specific features.

• Debriefing session in which experts work

As the experts work through the scenario

• Performed by a carefully managed team.

• Times of day &

• Less expensive than user testing.

• Usefulness limited to systems with

• Based on expert error-free behavior.

• Design guidelines can be used to develop heuristics

• Walkthroughs are focused so are suitable for evaluating small parts of a

• Analytics involves collecting data about users activity on a website or

You might also like