The domestication of

touchscreen technology and

interfaces and itʼs effect on
visually-impaired individuals.
by
Will Jenkins
# 6208463

Submitted to
Professor Tonio Sadik
for the course
Technologies, World and Society
(SOC 3116C)

University of Ottawa
November 15, 2010
In todayʼs developed economies, the value of information is vastly greater than it has been in

the past. A movement toward innovation in the development of knowledge and other non-

physical assets has coincided with the rapid adoption, diffusion, and domestication of a

variety of Information and Communication Technologies (ICTs). Therefore, the ability to

access and create information has become a central factor in discriminating between social

groups and classes. As stated by Karshmer (1995, pg 1), “Access to the information highway

may prove to be less a question of privilege or position than one of the basic ability to function

in a democratic society”.

Many technologies develop through the ABC of social power (Green 2002) - the military,

government and corporate establishments - before reaching the mass market and becoming

available to consumers. This tends to mean that technologies are designed for specific uses,

and are then adapted to meet the needs of the mass market by enterprises. Unfortunately

however, this can often mean that users whose abilities and requirements donʼt conform to

those of the “mass market” may be unable to access, use, or fully realise the potential of new

technological developments, for example users with all types of disabilities. This evolutionary

process also means that technology is out of step with other areas such as architecture,

education, or employment, which in various countries have regulations ensuring that people

with disabilities arenʼt discriminated against in terms of accessing products and services

provided.

The effect of this misalignment is that these users are unable to fully access the information

available to the majority of users and are put an an unfair disadvantage in their pursuit of work

and leisure activities (Glinert and York 2008). One group in particular being disadvantaged by

this off-kilter technological evolution is visually impaired users, specifically low-vision or zero-

1! SOC3116, Fall Semester 2010, Student #6208463

vision individuals. The importance of symbolic communication to the human experience

cannot be understated, and visually-impaired people have significantly reduced abilities to

perceive much of the communication that fully-sighted users take for granted, so this needs to

be counterbalanced in other ways. Especially when technology mediates communication, it

can be difficult for those with visual problems to understand the depth of communication that

is occurring because of a lack of consideration of their abilities and requirements in the

development and implementation of new technology.

One emerging technology which is rapidly being adopted by the mass market is the use of

touchscreens and related user interfaces in a wide variety of devices, replacing previously

tactile devices and interfaces. Led by the latest wave of smartphones including Appleʼs

iPhone and devices running on Googleʼs Android platform, touchscreens are being used in a

broader array of technologies than ever including cameras, computers, public information

kiosks and ATMs. Devices which use touchscreens are being trumpeted in their markets as

providing a new paradigm for user interaction which is easier to understand, more flexible,

and is fundamentally reshaping the way society thinks about how computers should work (for

example shifting from using many applications at once to using only one). This marketing

strategy has certainly been successful, with over 120 million devices running Appleʼs iOS and

Googleʼs Android having been sold over the past few years (AdMob Inc. 2010).

One serious and unanticipated problem with the adoption of touchscreen technology,

however, is the effect it has on the ability of visually-impaired individuals to use the devices

harnessing this ʻinteraction revolutionʼ (Mathema 2009). As McGookin (2008, pp 1-2) states,

“Unlike the controls on a standard mobile telephone or public access terminal, that can be

2! SOC3116, Fall Semester 2010, Student #6208463

easily felt through touch, touchscreen technologies do not provide any tactile distinguishment

between controls and display space. Whilst a visually impaired person can learn the locations

and functions of tactile control panels on current mobile telephones and public access

terminals, attempting to do the same with touchscreen based devices is much harder, due to

the lack of tactile distinguishment between virtual buttons and surrounding surfaces.”

Criticism can also be leveled at past touch-based devices, which due to to the technology in

the screens were imprecise, inconsistent and generally difficult to use - even for individuals

with perfect vision. For many visually impaired users, having these awkward touchscreens

domesticated into their lives has left a bad impression of touch technologies that still exists

today, even though the experience of using such devices has progressed so significantly. If

the touchscreen is on something that the user interacts with every day, for example a

microwave at home, a workaround will often be developed such as tactile annotations on the

screen or simply memorising the locations of buttons. However, if the touch interface is

something unfamiliar (for example an ATM that the person has never been to before), a blind

user will often have to either ask for help, or will simply avoid using the machine (Kane,

Bigham et al. 2008).

This is not to say that tactile interfaces have been ideal in the past. McGookin (2008) revealed

that while many specifically designed assistive technologies are available, they are often very

expensive and badly-designed and so users will instead appropriate “mainstream” devices to

their needs rather than using assistive devices. For example, the choice between a

specifically-designed but expensive mobile phone which allows access to many features in

the one device compared to a cheap, yet large and simple mobile phone where only basic

tasks can be accomplished is one that visually impaired users must often make. Users will

3! SOC3116, Fall Semester 2010, Student #6208463

also choose devices based on how good they are at a particular task and how easily that task

can be completed with a device, rather than getting an all-in-one device that is difficult to

operate or using the functionality already built into something they already own.

It is also important to consider the effect of software upon computer interactions, and the

trends which occur between hardware and software. Specifically, visually impaired users are

most affected by the alignment of mouse-based with desktop and touch-based with mobile

systems.

The traditional WIMP (windows, icons, menus and pointers) interface that most major desktop

computer operating systems use is a relatively complex paradigm, even for users with full

vision. Nonetheless it has been domesticated as the standard interaction scheme for desktop

computers, and most users have some degree of familiarity with itʼs operation. However,

because low- and zero-sight users are less able to gather visual information, and thus find it

much harder to distinguish small objects like icons and pointers, interaction with WIMP-based

systems becomes very difficult (Fraser and Gutwin 2000). It is also difficult for users with

limited vision because the mouse provides no frame of reference against which to position the

cursor on the screen, making it easier to “lose” the cursor and more difficult to make accurate

selections and movements. Nonetheless, because the WIMP model has become the standard

interface, many assistive tools have been developed to supplement its shortcomings,

including screen-reading technology, Braille “monitors” and magnification and colour

adjustment software. As Glinert and York (2008, pg 4) state, “What is required are

mechanisms for fundamentally modifying the visual aspects of the display in conjunction with

the manner in which information is conveyed. In the absence of such mechanisms, modern

window-based interfaces are more often a curse than a blessing to handicapped individuals.”

4! SOC3116, Fall Semester 2010, Student #6208463

With some training and assistance, visually impaired users are able to become almost as

proficient as sighted users.

The prevailing interaction scheme used in touch-based devices today is a task-focused one (a

modal interface) where in an interface a user can perform only one task and can only exit that

interface by either completing the task at hand or cancelling it (Vanderdonckt and Berquin

1999). This concept developed through a combination of technological necessity (mobile

devices have limited screen space and so must present information in the most efficient and

effective way) and usability innovation (having only one task on screen makes the device less

confusing to use). Because touchscreens directly link the user input with the output (for

example pinching a photo to zoom out or tapping a toggle to switch a feature on and off),

fully-sighted users generally find touch-based devices more intuitive upon first encountering

them. This also eliminates a key difficult of other interfaces - the mouse. By providing this

direct interaction, there is no requirement for a point of reference against which to manipulate

the device (other than the device itself) . Visually impaired users do however experience the

same frustrations with touchscreens/task-based interfaces in terms of deciphering information

as they do for mouse/WIMP interfaces - namely, their lack of visual acuity makes it difficult to

determine what they are supposed to be touching or gesturing on (Kane, Bigham et al. 2008).

What this also means is that interfaces between different tasks and pieces of software can be

vastly different, unlike the relatively uniform and predictable designs of WIMP interfaces which

must conform to the control scheme enforced by the accepted keyboard and mouse

combination. (Bonner, Brudvik et al. 2010)

The main problem for visually-impaired users caused by the introduction of touchscreens is

that the accessibility features of the devices in which they are used are significantly

5! SOC3116, Fall Semester 2010, Student #6208463

underdeveloped, in comparison with the accessibility features of tactile interfaces (Bonner,

Brudvik et al. 2010).

Accessibility in touchscreen interfaces is an issue that is presently being addressed by

hardware and software developers. As noted before, there is presently no mandated

requirement for accessibility in technological devices, but manufacturers are realising that

catering to people with visual impairments will increase the potential size of the market they

are able to sell to, as well as improving the interfaces they design for fully-sighted users - that

is, by adding accessibility features they can make their devices more useful to all potential

customers. This completes the technological-determinist/social-constructionist feedback loop,

by making the naturally developed technology of touchscreens fulfill the social requirements

which are also placed upon it.

The accessibility feature of touchscreens follow the same logic as the features in tactile

interfaces; that is, ʻsensory substitutionʼ, whereby the same information is conveyed in a

different mode so that it is perceived by another sense (in this case, a non-visual sense like

touch or hearing). The general concept of cross-modal icons seems to be an effective one,

where touching a visual icon on screen results in unique earcons and tactons (aural and

tactile equivalents to a visual icon) which represent the area or item touched (Hoggan and

Brewster 2007). This feedback scheme can be taken even further, with the use of intelligent

speech synthesis and voice recognition so that devices can generate on-demand audio for

reading out web pages or other ad-hoc text and displayed information, and can respond to a

userʼs natural language inputs. Both Appleʼs iOS and Googleʼs Android have had the features

recently incorporated into their software in limited ways, along with modified touch interaction

protocols to make manipulation by visually-impaired users more effective (DisabledWorld.com

6! SOC3116, Fall Semester 2010, Student #6208463

2010) . This interaction between between accessibility features and a touch interface

demonstrates the largest potential benefit for visually-impaired users:

“Because VoiceOver (iOSʼs inbuilt screenreader) works with iPhoneʼs touchscreen, you

interact directly with objects on the screen and can naturally understand their location and

context. So, when you touch the upper-left corner of the screen, youʼll hear whatʼs in the

upper left corner of a web page, and as you drag your finger around the screen, youʼll learn

whatʼs nearby, providing an amazing new sense of context and relationship between the

items you hear. For many, VoiceOver on iPhone will provide, perhaps for the first time, a true

sense of how things appear on screen, not just descriptions of what they are.” (Apple Inc.

2010).

At present, significant research is occurring into the most effective way to further develop

accessibility features for visually-impaired users to aid their interaction with touch-based

devices, and a number of recommendations can be drawn from various literature:

- Accessibility features need to be further developed as a deep part of the software that touch-

based devices run so that all programs on the device can access uniform, sophisticated

accessibility frameworks.

- Users with different types of visual impairment have different requirements in terms of

feedback modalities and presentations, and these must be considered.

- Uniformity in interface design is required, that is certain actions always have the same

feedback or are always activated in the same place so that users can transfer knowledge

between applications. Cross-modal feedback is also required for all interactions if the user

requests it.

7! SOC3116, Fall Semester 2010, Student #6208463

- Users must have a tactile point of reference in relation to the screen, for example the

“Home” button on Appleʼs iPhone. Likewise, avoid using interfaces which rely on spatial

awareness to activate features because this is often a stumbling-block for visually impaired

users.

Touchscreen technology seemingly has significant potential to positively alter the way visually

impaired users access and create information. In a society and economy that is based around

knowledge as a commodity, this it a very valuable opportunity. At present the evolutionary

mismatch between touch devices and their relevant accessibility features reduces the utility of

touch devices, however with increased awareness of the plight of disabled users along with

the rapid adoption and domestication of touchscreens, these features are being swiftly

developed to match the advanced state of touchscreen hardware. It is also important to note

that improved accessibility for visually-impaired users helps to develop better visual and non-

visual interfaces for sighted users, and the knowledge that is attained here in terms of

interaction modes and methods can be applied across a broad spectrum of areas and will in

the future help to achieve universal information accessibility regardless of ability.

8! SOC3116, Fall Semester 2010, Student #6208463

Bibliography

Academic Sources
!
Bonner, M., J. Brudvik, et al. (2010). No-Look Notes: Accessible Eyes-Free Multi-touch Text
Entry. Pervasive Computing. P. Floréen, A. Krüger and M. Spasojevic, Springer Berlin /
Heidelberg. 6030: 409-426.
!
Fraser, J. and C. Gutwin (2000). A framework of assistive pointers for low vision users.
Proceedings of the fourth international ACM conference on Assistive technologies.
Arlington, Virginia, United States, ACM: 9-16.
!
Glinert, E. P. and B. W. York (2008). "Computers and People with Disabilities." ACM Trans.
Access. Comput. 1(2): 1-7.
!
Green, L. (2002). Communication, Technology and Society. Sydney, Allen and Unwin.
!
Hoggan, E. and S. Brewster (2007). Designing audio and tactile crossmodal icons for mobile
devices. Proceedings of the 9th international conference on Multimodal interfaces.
Nagoya, Aichi, Japan, ACM: 162-169.
!
Kane, S. K., J. P. Bigham, et al. (2008). Slide rule: making mobile touch screens accessible to
blind people using multi-touch interaction techniques. Proceedings of the 10th
international ACM SIGACCESS conference on Computers and accessibility. Halifax,
Nova Scotia, Canada, ACM: 73-80.
!
Karshmer, A. I. and K. Kaugars (1995). "Equal access to information for all: making the world
of electronic information more accessible to the handicapped in our society." SIGCAPH
Comput. Phys. Handicap.(52-53): 11-23.
!
McGookin, D., S. Brewster, et al. (2008). Investigating touchscreen accessibility for people
with visual impairments. Proceedings of the 5th Nordic conference on Human-
computer interaction: building bridges. Lund, Sweden, ACM: 298-307.
!
Vanderdonckt, J. and P. Berquin (1999). Towards a very large model-based approach for user
interface development. User Interfaces to Data Intensive Systems, 1999. Proceedings.
!

Non-academic sources

AdMob Inc. (2010). AdMob Mobile Metrics: Metrics Highlights.

!
Apple Inc. (2010). "Apple - Accessibility - iPhone - Vision." Retrieved 10/11/10, 2010, from
http://www.apple.com/accessibility/iphone/vision.html.

DisabledWorld.com (2010). "Accessibility features in Android 2.0 Platform." Retrieved

15/11/10, from http://www.disabled-world.com/communication/voip/accessibility-
android-2.php.
Mathema, C. (2009). Multi-Touch All-Point Touchscreens: The Future of User Interface
Design. Mobile Dev and Design, Penton Media.