Understanding Shortcut Gestures On Mobile Touch Devices: September 2014

Download as pdf or txt
Download as pdf or txt
You are on page 1of 11

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/270887854

Understanding Shortcut Gestures on Mobile Touch Devices

Conference Paper · September 2014


DOI: 10.1145/2628363.2628378

CITATIONS READS
29 1,375

5 authors, including:

Niels Henze Wilko Heuten


Universität Regensburg OFFIS
223 PUBLICATIONS   4,632 CITATIONS    226 PUBLICATIONS   2,647 CITATIONS   

SEE PROFILE SEE PROFILE

Susanne CJ Boll
Carl von Ossietzky Universität Oldenburg
478 PUBLICATIONS   6,241 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Lumicons View project

Attention and Affective Computing Research View project

All content following this page was uploaded by Niels Henze on 23 December 2017.

The user has requested enhancement of the downloaded file.


Understanding Shortcut Gestures on Mobile Touch Devices
Benjamin Poppinga1 , Alireza Sahami Shirazi2 , Niels Henze2 , Wilko Heuten3 , Susanne Boll1
1 2 3
University of Oldenburg VIS, University of Stuttgart OFFIS - Institute for
Oldenburg, Germany Stuttgart, Germany Information Technology
firstname.lastname@uni- [email protected] Oldenburg, Germany
oldenburg.de stuttgart.de [email protected]

ABSTRACT smaller form factors and display sizes, touch-based interac-


Touch gestures become steadily more important with the on- tion runs into limitations as buttons and other interactive ele-
going success of touch screen devices. Compared to tradi- ments become too small to be operated accurately. Thus, al-
tional user interfaces, gestures have the potential to lower ternative interaction techniques have been invented and stud-
cognitive load and the need for visual attention. However, ied. Most observable, we saw a breakthrough of gesture-
nowadays gestures are defined by designers and developers based interaction on many recent mobile devices.
and it is questionable if these meet all user requirements. In
On nowadays devices a small set of simple touch gestures
this paper, we present two exploratory studies that investi-
is used for zooming, rotating, and scrolling. Further, touch
gate how users would use unistroke touch gestures for short-
gestures are also used to provide users with a shortcut to cer-
cut access to a mobile phone’s key functionalities. We study
tain functionalities, e.g., a swipe gesture to unlock the phone
the functions that users want to access, the preferred activa-
or to show the device’s camera. Typically, these gestures
tors for gesture execution, and the shapes of the user-invented
were defined by the designers and developers of a system
gestures. We found that most gestures trigger applications,
[13], although previous work showed that gesture sets that
letter-shaped gestures are preferred, and the gestures should
are mainly informed by the designers’ intuition might not
be accessible from the lock screen, the wallpaper, and the no-
satisfy the users’ requirements [23]. Beyond a basic set of
tification bar. We conclude with a coherent, unambiguous
well-established gestures, touch gestures for mobile phones
set of gestures for the 20 most frequently accessed functions,
are still not completely understood. For designers and devel-
which can inform the design of future gesture-controlled ap-
opers it is unclear which action a gesture should cause, how
plications.
the gesture should be shaped, and where the gestures should
be executed.
Author Keywords In this paper, we study unistroke touch gestures for instant ac-
Touch gestures; shortcut; mobile phone; gesture recognition; cess to a mobile phones’ core functions in the wild and from
instant access; quick launch. a user’s perspective. To do this we design our apparatus, Ges-
tify, which allows users to train and recognize custom ges-
tures on Android phones. We deploy this application on the
ACM Classification Keywords
Google Play store and observe how users create and use ges-
H.5.2 Information Interfaces and Presentation: User Inter-
tures in their real life. Further, we conduct a supplemental
faces—Input devices and strategies
lab study with 18 participants to substantiate these observa-
tions and to collect additional qualitative insights. We study
INTRODUCTION which actions users prefer to cause with their gestures, which
Over the past years, mobile devices, such as smart phones, shapes their gestures have, and how users trigger gestures in
tablets or wearable devices, started to replace traditional their everyday life.
desktop computers as the most successful computing devices. The contributions of this paper are as follows:
Many of these mobile devices use touch-sensitive displays
or surfaces as primary means of interaction, often mimick- • We show that gestures are a valuable means for instant
ing traditional user interfaces like keyboards. However, with access and they should be executable on phones’ lock
screens, notification bars, and wallpapers.

Permission to make digital or hard copies of all or part of this work for personal or • We highlight the 20 most important functions that should
classroom use is granted without fee provided that copies are not made or distributed be accessible through shortcut gestures.
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
• We identify that letter-shaped gestures, which represent the
republish, to post on servers or to redistribute to lists, requires prior specific permission first character of a related action, are used most commonly.
and/or a fee. Request permissions from [email protected].
MobileHCI 2014, September 23–26, 2014, Toronto, ON, Canada. • We provide a consistent gesture set for the 20 most impor-
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3004-6/14/09. . . $15.00. tant functions.
http://dx.doi.org/10.1145/2628363.2628378
In the next section, we provide an overview of previous work many fingers are used. Wobbrock et al. conclude with a user-
on touch and shortcut gestures on mobile devices. After- defined standard gesture set for common surface computer
wards, we describe the apparatus that we used to conduct two tasks and a taxonomy of surface gestures. Seyed et al. ap-
users studies. We describe an ‘in the wild’ field study to ob- plied a similar approach for multi-display environments [20].
serve users’ real life behavior, and a controlled lab study to
substantiate the findings. We discuss the results of the studies Ouyang and Li explored user-defined gesture shortcuts on
and derive a consistent gesture set for the 20 most important mobile phones to realize a crowdsourced gesture recognition
functions. Finally, we conclude and provide potential direc- system [16]. They explored user-defined gesture shortcuts of
tions for future work. 26 Android users and found that about 72 % of all gestures
were of alphanumeric nature, i.e., a character or letter. They
observed that users often define similar gestures for different
RELATED WORK actions, resulting in ambiguity and a decrease of recognition
Gestures have a long history in human-computer interaction rate to about 70 %. Further, they reported that only a few ges-
(HCI). They are one facet of interaction techniques that are ture actions were shared by all users. Thus, in most cases the
sometimes dubbed natural user interfaces [22], and their ex- users defined gestures for very individual actions.
ists a large body of work which investigated a broad range of Another interest for the community lies in the application
different gestural interfaces (see [4] for an overview). and everyday use of gestures [11, 21]. Gesture Search by
Previous work mainly investigated three different types of Li [11] allows users to search their mobile phone for apps,
gestural interfaces: free-hand gestures in mid-air, gestures as contacts, and media files with the help of letter-shaped stroke
alternatives for traditional user interface elements, and touch gestures. Thereby, a variety of actions can be controlled, e.g.,
gestures. Work on free-hand gestures that are executed in application launches, phone book access, or music playback.
mid-air most closely resemble gestures that humans use for The research focuses on how the gesture recognition can be
communicating with each other [5]. With the release of the combined with traditional user interface elements. They con-
Wii and the Kinect, free-hand gestures became widely used cluded that Gesture Search enables a quick and easy way to
for gaming and different applications in HCI research [19]. In access mobile data and users appreciate the usefulness and
desktop computing, gestures were mostly investigated as al- usability.
ternatives or complements for direct manipulation interfaces. In summary, most of existing work provides valuable insights
Moyle and Cockburn, for example, analyzed mouse and pen into a single research question, e.g., what do users want to
flick gestures [14]. With the recent success of touch screen control via gestures, how should a gesture look like, and how
devices, touch gestures became the most widely used type of can gestures be applied in various settings or scenarios. Few
gestural interface and a focus in HCI research. papers study gestures in context or in the wild, and come with
Touch gestures are meaningful strokes which are executed on other limitations. For example, Gesture Search [11] was lim-
a touch-sensitive surface, like a mobile phone, a tablet or a ited to letter-shaped gestures and their quick start application
larger tabletop computer (see [26] for a review). It has been wasn’t accessible from anywhere in the operating system. In
shown that touch gestures have a similar performance as key- contrast, we follow an exploratory approach and investigate
board shortcuts, but come with significant cognitive advan- the overall nature of touch gestures for shortcut access on
tages, i.e., they are easier to learn and recall [3]. In addition, mobile phones. With this approach we derive a coherent ges-
stroke gestures can be used eyes-free and are particularly ben- ture set, which combines our findings on the gesture actions
eficial for selecting frequently executed commands [17]. The and ideal gesture shapes, that should be accessible through a
variety of different application domains, settings, and tech- phone’s lock screen, notification bar, and wallpaper.
niques is huge, e.g., [8, 6, 12, 20]. The most of these domains
and applications are investigated on a by-case basis. Conse-
APPARATUS: GESTIFY
quently, a practical, coherent understanding of touch gestures
is hardly existing and no comprehensive guidelines for gestu- To investigate how touch gestures as shortcut to the phone’s
ral control are available [25]. functionalities are used, we developed the Gestify applica-
tion, that enables users to define gestures and use them to
Previous work not only investigated the design of touch ges- access various functions of their phone. In the following, we
tures but also explored design processes for gestural inter- describe the design of the application, the used gesture recog-
faces. In general, it has been found that user-defined gestures nition algorithm and the data that is recorded.
are easier to memorize than pre-designed gestures [15]. Ac-
tual designs for user-defined gestures are typically identified
in participatory design studies [23, 10, 9, 20]. Most promi- Creating and Managing Gestures
nently, Wobbrock et al. [23] presented an approach to de- Gestify is an Android application, running on Android 3.0 or
sign tabletop gestures with the help of potential users. They newer, that consists of two sub applications: Gesture Man-
showed the actual effect a gesture would cause and then ask ager and Gesture Recognizer. With installation of the appli-
the user to define an appropriate action, i.e., gesture. In a lab cation both sub applications are installed and added to the
study with 20 participants they recorded gestures, performed home screen. The Gesture Manager allows users to manage
with one or two hands. They found that users prefer to exe- existing gestures and define new ones, the Gesture Recog-
cute gestures with one hand and mostly do not care about how nizer does the actual gesture recognition.
(a) List of existing ges- (b) Add a new gesture. (c) Trigger detection (d) Draw gesture in recog- (e) A pop-up indicates that
tures. through one of the pro- nition activity. the gesture is detected.
vided activators, e.g.,
notification bar.
Figure 1. Gestify is an Android application that allows users to define and execute their own shortcut gestures. If gestures were trained to the system
they can be recognized via different activators, i.e., through the lock screen, the notification bar, the wallpaper, and through a separate activity. If a
gesture is detected, a toast is shown and the linked action is executed.

When the Gesture Manager is launched, an initially empty Using Shortcut Gestures
list shows the names of all existing gestures (see Figure 1a). The second sub application, Gesture Recognizer, is the inter-
A new gesture can be created by pressing the add button in face for the actual gesture recognition. The app includes a
the top right menu. The subsequently appearing dialogue is single, full screen view in which the user can draw a gesture.
shown in Figure 1b and first asks the user to name the new A drawn gesture is taken as input for the gesture recognition
gesture. The name can consist of alphanumerical characters engine and, based on the gesture recognition result, the cor-
and is recorded solely to allow users to maintain an overview responding action assigned to the gesture is executed. The
over existing gestures (as shown in Figure 1a). view can be reached through four different ways, namely, (1)
the lock screen, (2) the phone’s notification bar, (3) directly
Gestify only supports unistroke gestures as defined in [26].
launching the Gesture Recognizer app, and (4) a live wall-
Thus, the touch movement for defining a gesture has to be
paper. In the following, we refer to these different ways to
a single, continuous movement that can have a theoretically
trigger the gesture recognition view as activators.
unlimited complexity and length. Consequently, a single tap
or two successive strokes are no valid gestures. Although By installing Gestify and respectively the Gesture Recognizer
there are multistroke gesture recognition engines available, app, the gesture recognition view automatically replaces the
e.g., [2], we decided for a unistroke recognition to emphasize device’s current lock screen. Further, this view can be reached
the shortcut characteristic of the gestures. through a short cut in the Android notification bar (see Fig-
ure 1c). The lock screen and notification bar activators are
In the last step of the creation process an action needs to be
enabled by default, although they can be individually dis-
defined, which is triggered once the gesture is recognized.
abled through the application settings. In addition, the ges-
The user can select to start any application that is installed
ture recognition view can serve as a live wallpaper, which
on his or her phone (application gestures), show a contact
allows to recognize gestures straight from the phone’s home
from the phone book (contact gestures) or change a system
screen. Due to technical restrictions users need to manu-
setting (setting gesture). While any application or contact can
ally enable the wallpaper activator. To avoid confusion with
be selected, only core system settings, i.e., Bluetooth, Wi-
other wallpaper-related interactions, e.g., changing the home
Fi, Ringer, Vibration, can be changed for technical reasons.
screen with a sideways swipe, the gesture recognition needs
The user can decide to enable, disable or toggle one of these
to be initiated through a double tap onto an empty area. A
system settings. A tap on the OK button initiates the actual
pop-up briefs the user about this procedure.
gesture training, closes the dialogue, and guides the user back
to the gesture overview. Visual feedback is provided while drawing a gesture (see
Figure 1d) to increase the drawing accuracy [1]. The ac-
To simplify the analysis of recorded gestures, once created, a
tual recognition is triggered once the finger is lifted from the
gesture cannot be updated. After a gesture has been designed
screen. The result for each recognition trial is displayed as
and associated with an action, neither the name, the shape
a pop-up, which either shows the gesture label and recog-
nor the action can be modified. If the user wants to change a
nition probability or that no gesture could be detected (see
gesture, it must be deleted and a new one has to be created.
Figure 1e). If a gesture is recognized, the according action is
immediately executed.

Gesture Recognition Engine


Abstract Geometric Icon Letter Word
Because the user should not be asked to provide several train-
Figure 2. We empirically identified five different shapes each gesture can
ing gestures of the same type, we decided for a gesture recog- be assigned to. This figure shows an example for each of the shapes.
nition engine, which achieves high accuracies for few sam-
ple sizes. Gestify uses the $1 recognizer by Wobbrock et al.
[24], which is based on geometric template matching, and 1 to a maximum of 25 gestures. On average, each user trained
achieves recognition rates of over 97 % with a single training 2.92 gestures (SD 3.61, median 1) and used the application
instance. Although we did some empirical testing on the indi- for 5.15 days (SD 22.10, median 0.002). Given these numbers
vidual parameters of the engine, we eventually went with the it is clear that the majority of users just tested the application
best-performing default configuration [24]. That means every for a short time and then stopped using it.
gesture is scaled to a 500x500 pixel square and consists of 64
points. The gesture recognition engine has some limitations, Application Categorization
such as the inability to deal with ambiguous gestures. These In total, 303 unique applications were launched using Ges-
limitations were not communicated to the users to avoid lim- tify. To investigate for which application category the gesture
iting them in their creativity. action was used, we retrieved application categories from the
Google Play marketplace. However, as also reported by Sa-
Data Logging hami et al. [18], we realized that the categories are too generic
The Gestify application is accompanied by a custom logging to differentiate between application functionalities. Further,
framework. Once the application is installed, various device also the class/package names are inconsistent, rarely self-
information are recorded, e.g., the manufacturer, the model explaining and, thus, don’t allow for a classification. Con-
and the locale. Further, the logging framework records each sequently, we manually investigated the underlying function-
created gesture, including their label and action, and when a ality of each application and assigned it to a unique and il-
gesture is deleted. All recognition attempts are stored with lustrative category. Thereby, we followed the approach by
their origin, i.e., wallpaper, lock screen, notification bar, or Sahami et al., but successively added more categories, when
the activity itself, and their recognition result. For ethical we realized this was necessary. This resulted in 17 categories.
reasons no personal information, e.g., the contact name for Categories had between 4 and 155 unique applications (mean
gestures which trigger a phone call, is logged. The log data is 17.82, SD 35.80, median 7).
transferred to a central server every two minutes.
Gesture Shape Categorization
‘IN THE WILD’ FIELD STUDY
Gestify allows a user to use any type of gesture shape. For a
In our first study, we investigate how people use the Gestify better characterization and discussion of the various shapes,
app ‘in the wild’, i.e., in their everyday life. We determine the we manually analyzed and classified all gestures in a two-
actions that are triggered, the gesture shapes that are drawn, step process. In the first step, we went through all gestures
and the activators which are used to do the gesture recogni- and identified a set of five shape classes: abstract, geometric,
tion. icon, letter and word shapes (see Figure 2). Abstract gestures
consists of not inherently meaningful, not closed movements,
Recruitment
e.g., a straight swipe or a wave-like movement. Geometric
We published Gestify in the Google Play store in April 2013. gestures look like well-known, closed geometric shapes, e.g.,
Until January 2014 the application was downloaded and used a circle, a square, or a heart. Icon gestures recap parts of an
by 388 users. The majority, 259 users (66.75 %) had an En- app-icon or other well-known icons, e.g., an envelope (for a
glish locale, e.g., en US, en GB followed by German locales messenger application) or a pentagram. Letter gestures look
(de DE) with 24 users (6.19 %). In total, 124 users (31.96 %) like an alphabetic character or digit, and word gestures con-
had an European/African time zone, i.e., GMT/0 to GMT/2 tain more than one letter or whole words. In the second step,
and 70 (18.04 %) an American timezone, i.e., GMT/-4 to we went through all gestures again and assigned each of them
GMT/-8. This shows that our application was mostly used to a single category. In the majority of all cases the classifica-
by English- or German-speaking EU citizens and US Ameri- tion was obvious. In borderline cases we assigned a gesture
cans, which has to be considered when interpreting the results to the more generic class, e.g., an f -shaped gesture for the
[7]. All users were using phones with Android 4.0 or newer. Facebook application was assigned to the letter category and
not to the icon category.
Dataset
Gestify recorded information about the device, trained ges-
‘Power Users’ Subsample
tures and gesture recognition attempts. This data was trans-
ferred over the Internet and stored on a server. It was analyzed To obtain a better understanding about how gestures are used,
using the R software environment for statistical computing. we extracted a sub-sample of users who used the application
for a longer period. This allows us to study if the gesture
Our analysis shows that the 388 users trained 1134 gestures, actions, shapes, or activators differ between short-term and
whereby the number of trained gestures per user ranges from long-term users. We call these users power users and include
n All Users Power Users n

70
471 Other 41.53 % Other 44.66 % 113

60
111 Messenger 11.50 % Messenger 13.76 % 30 All Users
82 Phone 8.50 % Phone 10.55 % 23 Power Users

50
75 Browser 7.78 % Social Netw. 6.42 % 14

40
Percent
71 Camera 7.36 % Music Player 5.96 % 13

30
58 Music Player 6.01 % Camera 5.96 % 13
54 Gallery 5.60 % Mail 5.50 % 12

20
52 Social Netw. 5.39 % Gallery 4.59 % 10

10
41 Mail 4.25 % Browser 3.21 % 7
27 Settings 2.80 % Settings 2.29 % 5

0
Table 1. The top ten categories of application gestures. Most gestures Letter Abstract Geometric Icon Word
were defined for messaging apps, such as WhatsApp or SMS.
Figure 3. Gesture shapes for all users and power users. Letter-shaped
gestures are the most prominent among both user groups, observed in
about half of all cases.
only users who actively used the application for at least a full
week, i.e., 7 days, and trained and used at least 1 gesture. This
resulted in a set of 36 users, which closely resemble the orig- From 82 setting gestures observed 9 were identified as erro-
inal sample characteristics, i.e., locale and time zone. These neous and therefore were excluded from analysis. Of the re-
power users trained a total number of 253 gestures. Each maining 73 gestures, 17 (23.29 %) were configured to toggle
power users trained on average 7.03 gestures (SD 5.13, me- Wi-Fi, 14 (19.18 %) toggled Bluetooth, and 8 (10.96 %) en-
dian 5.5) and used the application for 52.72 days (SD 53.18, abled vibration. On average, each user uses a single gesture to
median 29.79). toggle a system settings, i.e., turn on or off a setting depend-
ing on its current state. About 89.04 % of all gestures were
Results configured to control Wi-Fi, Bluetooth, or vibration (about
In the following, we present the user-defined actions and vi- 30 % each). Only 10.96 % were used to change ringer set-
sual shapes of gestures. Further, we study which activator tings.
users preferred to use and draw their gestures on. We do this We made slightly different observations for power users.
analysis for both, all users and the identified ‘power users’ They contribute 19 setting gestures in total, of which none
subsample. was invalid. Of these, 6 gestures (31.58 %) were config-
ured to enable vibration. 5 gestures (26.32 %) toggled Wi-Fi
Gesture Actions
and 3 gestures (15.79 %) disabled vibration. 2 gestures each
Gestify differentiates between three types of possible ges-
(10.53 %) were used to toggle Bluetooth or vibration. A sin-
tures: start an application, show contact details, or modify
gle gesture (5.26 %) was set up to toggle the ringer. Similar
a system setting. Of the 1134 recorded gestures, 965 gestures
to the set of all users, the majority of gestures (94.74 %) op-
(85.10 %) were application gestures. 87 gestures (7.67 %)
erates Wi-Fi, Bluetooth, or the vibration actuator.
were contact gestures, and 82 gestures (7.23 %) were setting
gestures. Power users contributed 253 gestures, of which 218 Further, we observed 87 contact gestures for all users and 16
(86.16 ) are application, 19 (7.51 %) settings, and 16 (6.32 %) contact gestures for power users. However, for ethical rea-
contact gestures. sons we did not record the meaning of any contact gestures,
as this would allow us to track intimate contact details.
An analysis of the application categories shows that messag-
ing applications, such as WhatsApp or SMS/MMS messen- Gesture Shapes
gers, were the most frequently launched applications (111 Of the 1134 recorded gestures, letter-shaped gestures were
gestures, 11.50 %). Phone-related apps, such as the phone the most frequent ones with 546 gestures (48.15 %) (see Fig-
book or the dialer activity were the second most frequent ap- ure 3). 407 gestures (35.89 %) matched the abstract shape,
plications (82 gestures, 8.50 %) followed by browser applica- consisting of mostly simple swipes. The geometric shape
tions (the Android default browser, Chrome or Firefox) with included 99 gestures (8.73 %), the icon shape 48 gestures
75 gestures (7.78 %) of the total 965 application gestures. (4.23 %) and the word shape 34 gestures (3.00 %). We ob-
For power users a similar trend was observed, except that the served a similar distribution for power users, although the
third-most often triggered functionality were social networks
power users tend to have less abstract and more letter-shaped
instead of browsers (14 of 218 gestures, 6.42 %). About one
gestures.
third of all application gestures (302 of 965, 31.30 % and 78
of 218, 35.78 % for power users) were not of a particular pop- For a more in-depth analysis of the gesture shapes, we ex-
ular functionality, and therefore assigned to the ‘other’ cate- tracted a set of gestures for the most commonly actions used.
gory. For example, these gestures were used to trigger spe- The set consists of the top 3 applications from the largest cat-
cific games or niche applications, which are of importance for egory, i.e., other applications (Play Store, YouTube, Google
the individual user, but not of general relevance. An overview Maps), the top 2 applications from the second largest cat-
of the top ten actions for both user groups can be found in Ta- egory, i.e., messenger applications (WhatsApp, SMS Com-
ble 1. poser), and the first application of every other category with
n 39 37 32 29 27 26 25 24 16 15 13 13 13 13 13 11 5 2 87 17
Field Study

1st
31% 14% 28% 34% 81% 23% 28% 29% 38% 60% 38% 31% 23% 31% 31% 36% 20% 50% 9% 71%

2nd
18% 11% 9% 21% 4% 12% 12% 13% 19% 13% 31% 15% 15% 15% 23% 18% 20% 50% 8% 12%

Google Maps

Fav. Contact

Toggle Wifi
Phonebook

WhatsApp

Calculator
Play Store

Flashlight
Facebook

Launcher
Calendar
Browser

Youtube
Settings
Social Network

Camera

Gallery
Action

Music Player
Music
Clock
Messaging

Messaging

Calculator
SMS

Flashlight
Organiser
Mail

Launcher
Browser

Settings
Camera

Gallery
Others
Phone

Clock
Other

Other
Mail
1st
59% 15% 39% 24% 61% 33% 21% 29% 28% 71% 28% 28% 47% 44% 41% 28% 28% 28% 30% 22%
Lab Study

2nd
18% 15% 22% 18% 6% 28% 16% 24% 22% 12% 17% 22% 18% 17% 18% 22% 17% 17% 24% 11%
n 17 18 18 17 18 18 18 17 18 17 18 18 17 18 17 18 18 18 17 18
Table 2. A visualization of the top two gestures for the twenty most popular gesture actions. For some actions the gestures are very consistent among
users and studies, e.g., with about 71 % in the field study and 61 % in the lab study most people agree that WhatsApp should be triggered with a W
gesture.

70
more than 10 trained gestures. Further, we included the most
All Users
prominent settings gesture (i.e., Toggle Wi-Fi), and all con- Power Users
60
tact gestures. The resulting set consists of 20 representative 50

actions for the most commonly controlled functionalities and


is shown in Table 2.
40
Percent

Regarding the shape of application gestures, we found that


30

459 of the 965 gestures (47.56 %) were classified to have a


20

letter shape. Further, we identified 344 (35.65 %) abstract, 85


(8.81 %) geometric, 47 (4.87 %) icon, and 30 (3.11 %) word
10

gestures. The in-depth analysis shows that there is often a


0

strong consensus among users on which gesture shape should Lock Screen Notification Bar Activity Wallpaper
be used for which application (see Table 2). In fact, users
often select the first letter of the application name, e.g., W Figure 4. Users most frequently used the lock screen as activator for
to start WhatsApp or f to start f acebook. For power users the gestures recognition. The wallpaper activator was hardly used at all,
although study participants rated it best in the lab study.
we observed the same order with a similar distribution, i.e.,
56.77 % were letter gestures, 24.77 % abstract, 8.72 % geo-
metric, 5.05 % icon, and 4.59 % word gestures.
Again, the 9 erroneous setting gestures were excluded Gesture Activators
from the analysis. Of the remaining 73 setting gestures, Gestify offers four activators to start the gesture recognition.
35 (47.95 %) were of letter, 25 (34.25 %) of abstract, 9 A user could use the lock screen, the wallpaper, a separate
(12.33 %) of geometric, and 4 (5.48 %) of word shape. None activity or the notification bar. Overall, 12531 gesture recog-
of the gestures had an icon shape. Setting gestures of power nition attempts by 321 different users were recorded. Of
users were 78.95 % letter-shaped, 15.79 % abstract-shaped, these, 7263 (57.96 %) were performed on the lock screen,
and 5.26 % of geometric shape. The two most common ac- 3537 (28.23 %) via the notification bar, 1708 (13.63 %) via
tions for all users, i.e., toggle Wi-Fi and toggle Bluetooth, the activity, and 23 (0.18 %) on the wallpaper (see Figure 4).
were most frequently triggered with a w/W-shaped (12 of 17, A similar distribution was observed for power users. Overall,
70.59 %) respectively B-shaped (7 of 14, 50.00 %) letter ges- the lock screen seems to be the preferred activator to launch
ture (see Table 2). applications, change settings or open a contact.
We observed that 50 of 87 (57.47 %) contact gestures were We further studied which activator is most frequently used
letter shaped. 32 gestures (36.78 %) had an abstract, 4 per user. 135 users (42.06 %) most frequently used the lock
(4.60 %) a geometric, and 1 (1.15 %) an icon shape. No screen to execute a gesture. 130 users (40.50 %) used the
word-shaped gestures were observed. Power users created notification bar, and 56 users (17.45 %) used the activity most
12 (75.00 %) letter-shaped contact gestures, and 4 (25.00 %) frequently. None of the users used the wallpaper as their most
abstract gestures. frequent activator. On average, 85.56 % (SD 15.16 %) of a
users gesture recognition attempts were done using the same
activator. This indicates that most users prefer to permanently

compl.
agree
use a single activator instead of various activators in parallel.
6164 of 12531 gesture recognition attempts, i.e., 49.19 %

agree
(48.22 % for power users), led to a successful recognition of a
gesture. Thereby, the recognition rate varies between the indi-

neutral
vidual activators, i.e., 34.78 % for the wallpaper and 67.01 %
for the notification bar. We argue that the limited recognition
performance can mostly be credited to users, who studied the

disagree
limitations of the recognition, and accidental gesture recog-
nition attempts, e.g., caused by unintentional lock screen or
wallpaper use.

disagree
compl.
Lock Notif. Screen Back of Moving Arbitr.
Screen Bar Activity Wallp. Off Phone Phone App
LAB STUDY
The study through the marketplace was conducted in an un- Figure 5. Lab study participants were asked to provide their level of
controlled setting and its findings have high external validity. agreement to a set of possible gesture activators. The results show that
In the following, we complement these findings through a lab on average, participants mostly agree with performing gestures on the
wallpaper. The error bars show the standard error.
study, which will provide us with insights of internal validity.
We study if lab users of Gestify create similar shaped gestures
and what their preferred activators are. In the questionnaire we asked the participants to rate a state-
ment on each of the four different activators, e.g., I would
Method like to perform the gesture on the wallpaper, using a 5-point
We recruited 18 participants (7 female) with an average age of Likert scale from (1) ‘completely disagree’ to (5) ‘completely
26.4 years (SD 5.20). The participants were recruited through agree’. The analysis shows that 83.00 % of participants (mean
the university’s mailing lists. Twelve participants were stu- 4.16, SD 0.85, median 4) would agree to use the wallpaper
dents in different subjects such as Informatics, Electrical En- for gesture recognition. 77 % would agree to use the lock
gineering, etc. All participants owned an Android phone and screen (mean 4, SD 1.28, median 4.5). 83 % of the partic-
67 % used it for more than a year. None of the participants ipants disagreed with the statement on performing gestures
had used the Gestify application or another application with in a separate application (mean 1.83, SD 0.85, median 2)
similar functionality prior to the study. as it—according to received qualitative feedback—would re-
We installed Gestify on a Nexus 5 mobile phone and used it quire more interaction steps and, thus, is cumbersome. Fur-
during the study to collect data. After welcoming and signing thermore, 66 % disagreed with a gesture performance in the
the consent form, participants were asked to fill in a demo- notification bar (mean 2.27, SD 1.27, median 2).
graphic questionnaire. Then, we introduced Gestify through We further asked the participants to provide feedback on four
a short demo and explained its functionality. The participants other possible activators, i.e., the back of the phone, moving
were then asked to specify and train gestures for the 20 most the phone itself, drawing gestures on the phone’s screen when
popular actions, which we identified in the field study. The it is off, and draw a gesture while using arbitrary applications.
actions’ order was randomized to reduce sequence effects. The results reveal that 78 % of all users would agree to per-
Finally, we asked the participants to answer a questionnaire, form a gesture while using any arbitrary application (mean
which assesses the participants’ current and desired future use 4.05, SD 1.39, median 5). Further, 72 % agreed to perform
of activators for gesture shortcuts. Further, the questionnaire gestures while the phone screen is off (mean 3.77, SD 1.51,
provided room to leave qualitative feedback about the partici- median 4) and 56 % agreed with the back of the phone as an
pants’ impressions of using Gestify. The study lasted approx- activator (mean 3.44, SD 1.33, median 4). 56 % disagree with
imately 30 minutes. Each participant was compensated with moving the phone for performing a gesture (mean 2.44, SD
5e. 1.09, median 2). A visualization of these results is provided
in Figure 5.
Results
In total, 360 gestures were performed by the participants. The qualitative feedback shows that the participants consider
However, because of a technical failure 7 gestures could not simpler gestures for frequently used actions, since these are
be recorded. Consequently, our analysis is done with 353 ges- easier to memorize. Furthermore, it was mentioned that exist-
tures. To have an overview on the gesture shapes we reviewed ing symbols and icons, e.g., characters taken from the alpha-
all gestures collected for each action and assigned them to one bet, can be a good starting point for the definition of gesture
of the five shape categories, which we identified in the field sets.
study (see Figure 2). Based on these results, letter-shaped
gestures were the most frequent gestures (35.41 %), followed DISCUSSION AND RECOMMENDATIONS
by icon-shaped (26.91 %), abstract-shaped (26.62 %), geo- In the following, we discuss our results and provide recom-
metric-shaped (9.06 %), and word-shaped (1.42 %) gestures. mendations for designers and developers of future gesture
The two most frequent gestures for each action are shown in recognition systems on mobile touchscreen devices, such as
Table 2. mobile phones.
Various Activators to Enable Gesture Recognition Our results show that 85.10 % of all trained gestures in the
Gestify offered four ways to start the recognition process, i.e., field study were application gestures. Consequently, only a
via lock screen, notification bar, wallpaper, or a dedicated ac- minority of setting gestures and contact gestures were cre-
tivity. The field study indicated that the by far most frequently ated. This contradicts with earlier findings reported by Li
used gesture activator is the lock screen (credited for 58 % of [11], who observed that contact gestures (66 %) are preferred
all gesture executions, see Figure 4), followed by the notifica- over application gestures (28 %). Even applications from the
tion bar (28 %), the activity (14 %), and the wallpaper (very communication category and contact gestures together only
close to 0 %). The lab study results show that 83 % would sum up to 32.72 % of communication-related gestures, which
agree to use the wallpaper and 77 % would agree to use the is still half of the percentage that Li reported [11].
lock screen as gesture activator (see Figure 5). In contrast,
We argue that the reason for this difference lies in the slightly
83 % would disagree to use a separate activity as activator,
different interaction concepts. The app Gesture Search by Li
and 66 % would disagree to perform gestures in the notifica-
consists of a single activity, which needs to be started and
tion bar.
which then recognizes predefined character gestures. These
The lab study results indicate that users prefer the wallpa- are concatenated to words that serve as filter for a list of all
per activator, although the field study findings indicate that it available phone actions, such as ‘start application’ and ‘show
was hardly used in practice. A potential reason for this gap contact’. In contrast, Gestify can be accessed directly via
could be that the field study users were not fully aware of various activators, allowing to trigger an action with a single
the wallpaper activator, although it was equally well adver- user-defined gesture.
tised as other activators. Another potential reason could be
Thus, the omnipresence of gesture activators and the self-
that the technique to enable the activator, i.e., a double tap
contained, ‘single stroke’ character of a gesture seem to be
on empty space on the wallpaper, was too complex to create
two key criteria for appreciated shortcut gesture recognizers.
a real advantage over using the regular launcher. Further, it
This is supported by our lab study, where users preferred to
could have been confusing or misleading that also the regu-
execute gestures, e.g., on the wallpaper, in every application
lar home screen keeps reacting to touch input during gesture
or on the lock screen. All these activators are very present and
execution, which can lead to minor icon movement.
often just a button press away. Further, we hardly observed
Future gesture recognition systems should definitively respect word-shaped gestures in our studies. Instead, most users cre-
the desire for diversity and offer various ways to recognize a ated letter, icon or abstract gestures.
gesture. For mobile phones, gesture recognition should be
Designers and developers should consider these findings in
available through the lock screen, the notification bar, and
their future touch gesture recognition systems for shortcut ac-
through the wallpaper. However, because of our observations
cess. Future research could investigate which of the proposed
we recommend that the gesture interaction with the wallpa-
gesture activators perform best in everyday use. Further, also
per needs further research. For example, it should be clarified
the execution speed of various gesture types could be studied
how users exactly want an ideal gesture wallpaper to behave.
and how this affects the overall user experience.
Because we did not observe an intense use of the application
activator and because people don’t have a desire to use ges- Pre-defined Gesture Set
ture recognition in a separate activity, this option could be We observed that many gestures were created for a small set
omitted in future systems. of actions. In fact, we analyzed 1134 gestures from 388 users
We further identified that the preferred activator is user spe- to identify a set of 20 popular actions. This set is shown in
cific. In about 85 % of all gesture recognitions, users use their Table 2 and consists of 18 application gestures, one settings
most frequently used activator. For example, if a user’s most gesture and one contact gesture. We argue that this set is
frequently used activator is the lock screen, about 85 % of a good representation of the average user’s preferred actions
all gesture executions are done here. Thus, if for any rea- and, therefore, makes a good starting point for the definition
sons a user has to decide for a certain approach, most users of future shortcut tools for mobile devices.
will probably be able to make such a decision comfortably. Similar to Ouyang and Li [16], we observed that half of all
Future research could investigate if gesture actions for indi- 1134 gestures, i.e., 47.12 %, were letter-shaped. 35.89 %
vidual activators differ, e.g., if messaging gestures are more were of abstract shape, 8.73 % of geometric shape, 4.23 %
often triggered from the lock screen or the notification bar. of icon shape, and 3.00 % of word shape. This dominance
is also present within our top 20 set of actions (see Table 2).
Applications Are Most Popular Gesture Action
Qualitative statements from the lab study indicate that some
Gestify allowed users to use three types of gesture actions:
users used the alphabet and the action name as a reference for
application gestures to start applications, contact gestures to their gestures.
quickly reach for contacts, and settings gestures to modify
system settings. In fact, any application and any contact could Overall, our findings show a clear preference for letter-
be assigned to a gesture. Further, Gestify provides a function- shaped gestures. We argue that this is the case because they
ality, which—similar to existing quick settings applications— are powerful enough to represent and differentiate between
allows to instantly change system settings. Consequently, we advanced information, such as various apps. At the same time
argue that Gestify can be used to control these functionalities letters are simple enough to be memorized and remembered.
without any limitations. In contrast, abstract and geometric gestures have a limited
opers, although it is unclear if these gestures meet the end-
user requirements.
Facebook Phonebook Play Store Camera WhatsApp
In this paper, we studied the users’ needs and desires regard-
ing gestures from the ground up. We designed and developed
a flexible gesture recognition application, Gestify, which aims
Gallery Browser Settings SMS YouTube to fulfill most of the potential user needs. Users could define
their own gesture actions, gesture shapes, and use different
activators to initiate the gesture recognition. Using the Ges-
Mail Clock Calculator Google Maps Music tify application we conducted two exploratory user studies in
two different settings: a field study in the Google Play mar-
ketplace and a supplemental lab study with 18 participants.
Calendar Launcher Flashlight Toggle Wi-Fi Fav. Contact
Our results show that the majority, i.e., 85 %, of all analyzed
Figure 6. By analysing and combining gestures from both studies, we de- gestures are set up to control the user’s applications. About
rived a comprehensive gesture set for the top 20 actions. This gesture set
can serve as the default configuration for future mobile touch interfaces. 7 % each modify system settings or show contact information.
Half of all gestures, i.e., 48 %, have a letter shape, typically
mimicking the first letter of the controlled application name,
expressiveness, icon gestures can be too hard to remember, e.g., W for WhatsApp. 36 % have an abstract shape, repre-
and word gestures can be too complex to draw. senting, e.g., a single swipe. Our field study results show that
the lock screen (58 %) and the notification bar (28 %) are the
Both, the identification of common actions and the subse- most frequently used activators to trigger the gesture recog-
quent analysis of gesture shapes, allow us to design a co- nizer. However, the lab study reveals that the live wallpaper
herent and unambiguous gesture set. To derive this set we and back of the phone are also in the user’s preference for
analyzed all gestures that we observed in both studies. We triggering gestures.
used the absolute number of gestures, which were observed in
the field study, as a global indicator for importance. Starting We conclude that touch gestures are a valuable means for
from the most important gesture (left hand side in Table 2), instant access to a variety of actions, e.g., to start applica-
we calculated an overall percentage for each identified shape tions, to show contact details, or to toggle system settings.
by combining data from both studies. If the gesture shape We found that users prefer gesture activators, which are om-
with the highest percentage is not already used for a more nipresent and ideally less than a button press away. In fact,
important action, it was assigned to this gesture. Otherwise our results indicate that gesture execution should be possible
the next-most prominent shape is considered until an yet un- on the lock screen, on the wallpaper, and in the notification
used shape was found. This procedure allows us to consider bar. We observed that users rarely switch between various
both, i.e., how often a gesture was used, and which gesture activators. Instead, they decide for a single activator and con-
shape was used most frequently. tinue to use it. We argue that future gesture recognition sys-
tems should consider the need for diversity, but may ask users
The resulting set is shown in Figure 6 and could be used as to decide for a single activator if necessary.
a default working configuration, likely suiting the needs of
most users. A deployment of this set could allow users to We further highlight the 20 most important actions which
trigger actions without the need to train gestures in advance. should be accessible through shortcut gestures. Of these 20
This can positively affect the user experience and potentially actions the majority are application gestures, such as Face-
reduce the time to learn and understand the gesture recogni- book, WhatsApp, or the Camera application. One action each
tion application. Nevertheless, it must be clear that this set is dedicated to access contact information or toggle system
might also contradict the user’s desires. Therefore, it should settings. We identify that about half of all gestures are let-
still be customizable to ensure greatest flexibility and to ad- ter-shaped, and the second most prominent gestures are of
dress users’ individual needs. abstract shape. Overall, only a few gestures are of icon,
word, or geometric shape. Letter shapes typically represent
Future research should study how gesture actions, shapes, and the first character of a related action, whereby icon gestures
activators are designed for other user groups, e.g., users with typically mimic the application icon. Through our two com-
a non-Latin alphabet, and if and how the availability of a de- plementing studies we provide a comprehensive, unambigu-
fault gesture set affects the user experience. Further, it would ous gesture set for the 20 most important actions (see Fig-
be interesting to know if the importance of individual actions ure 6), which can serve as a customizable default gesture set
change over time, and if and in which cases users re-define a for future touch devices and thereby avoids time-consuming
gesture. gesture training efforts.
In our future work, we want to apply this proposed gesture set
CONCLUSIONS in the wild and study if users are satisfied or if they apply any
Touch gestures are becoming a more and more popular in- modifications. Further, we are interested to investigate how
teraction technique on nowadays smart phones, tablets, and gestures are used in everyday life. We plan to research which
wearables. Existing touch gestures, e.g., a sideways slide to gestures are used most frequently over the day, and if certain
unlock a device, are typically defined by designers and devel-
activators are preferred to execute certain gestures. Further, 13. Lü, H., and Li, Y. Gesture coder: A tool for
it remains questionable to what extend our proposed gesture programming multi-touch gestures by demonstration. In
set can be transferred to future mobile devices, like wearable Proceedings of CHI, ACM (2012), 2875–2884.
glasses or smart watches.
14. Moyle, M., and Cockburn, A. Analysing mouse and pen
flick gestures. In Proceedings of the SIGCHI-NZ
ACKNOWLEDGEMENTS
Symposium On Computer-Human Interaction, ACM
This work is partially supported by the German Research
(2002), 19–24.
Foundation within the SimTech Cluster of Excellence (EXC
310/1). 15. Nacenta, M. A., Kamber, Y., Qiang, Y., and Kristensson,
P. O. Memorability of pre-designed and user-defined
REFERENCES gesture sets. In Proceedings of CHI, ACM (2013),
1. Anthony, L., Brown, Q., Nias, J., and Tate, B. 1099–1108.
Examining the need for visual feedback during gesture
16. Ouyang, T., and Li, Y. Bootstrapping personal gesture
interaction on mobile touchscreen devices for kids. In
shortcuts with the wisdom of the crowd and handwriting
Proceedings of the 12th International Conference on
recognition. In Proceedings of CHI, ACM (2012),
Interaction Design and Children, IDC ’13, ACM (2013),
2895–2904.
157–164.
17. Roudaut, A., Bailly, G., Lecolinet, E., and Nigay, L.
2. Anthony, L., and Wobbrock, J. O. $n-protractor: A fast
Leaf menus: Linear menus with stroke shortcuts for
and accurate multistroke recognizer. In Proceedings of
small handheld devices. In Human-Computer
Graphics Interface (2012), 117–120.
Interaction – INTERACT 2009, vol. 5726 of Lecture
3. Appert, C., and Zhai, S. Using strokes as command Notes in Computer Science. Springer Berlin Heidelberg,
shortcuts: Cognitive benefits and toolkit support. In 2009, 616–619.
Proceedings of CHI, ACM (2009), 2289–2298.
18. Sahami Shirazi, A., Henze, N., Dingler, T., Pielot, M.,
4. Billinghurst, M. Gesture based interaction. In Haptic Weber, D., and Schmidt, A. Large-scale assessment of
input, B. Buxton, Ed. Cambridge University Press, 2009. mobile notifications. In Proceedings of CHI, ACM
(2014), 3055–3064.
5. Bolt, R. A. “put-that-there”: Voice and
gesture at the graphics interface. In Proceedings of 19. Schlömer, T., Poppinga, B., Henze, N., and Boll, S.
SIGGRAPH, ACM (1980), 262–270. Gesture recognition with a wii controller. In
Proceedings of TEI, ACM (2008), 11–14.
6. Döring, T., Kern, D., Marshall, P., Pfeiffer, M.,
Schöning, J., Gruhn, V., and Schmidt, A. Gestural 20. Seyed, T., Burns, C., Costa Sousa, M., Maurer, F., and
interaction on the steering wheel: Reducing the visual Tang, A. Eliciting usable gestures for multi-display
demand. In Proceedings of CHI, ACM (2011), 483–492. environments. In Proceedings of ITS, ACM (2012),
41–50.
7. Henze, N., Pielot, M., Poppinga, B., Schinke, T., and
Boll, S. My app is an experiment: Experience from user 21. Tran, J. J., Trewin, S., Swart, C., John, B. E., and
studies in mobile app stores. Int. J. Mob. Hum. Comput. Thomas, J. C. Exploring pinch and spread gestures on
Interact. 3, 4 (Oct. 2011), 71–91. mobile devices. In Proceedings of MobileHCI, ACM
(2013), 151–160.
8. Hinrichs, U., and Carpendale, S. Gestures in the wild:
Studying multi-touch gesture sequences on interactive 22. Wigdor, D., and Wixon, D. Brave NUI world: designing
tabletop exhibits. In Proceedings of CHI, ACM (2011), natural user interfaces for touch and gesture. Elsevier,
3023–3032. 2011.
9. Lahey, B., Girouard, A., Burleson, W., and Vertegaal, R. 23. Wobbrock, J. O., Morris, M. R., and Wilson, A. D.
Paperphone: Understanding the use of bend gestures in User-defined gestures for surface computing. In
mobile devices with flexible electronic paper displays. Proceedings of CHI, ACM (2009), 1083–1092.
In Proceedings of CHI, ACM (2011), 1303–1312.
24. Wobbrock, J. O., Wilson, A. D., and Li, Y. Gestures
10. Lee, J. C., Hudson, S. E., and Tse, E. Foldable without libraries, toolkits or training: A $1 recognizer
interactive displays. In Proceedings of UIST, ACM for user interface prototypes. In Proceedings of UIST,
(2008), 287–290. ACM (2007), 159–168.
11. Li, Y. Gesture search: A tool for fast mobile data access. 25. Wolf, K., McGee-Lennon, M., and Brewster, S. A study
In Proceedings of UIST, ACM (2010), 87–96. of on-device gestures. In Proceedings of MobileHCI,
ACM (2012), 11–16.
12. Lü, H., and Li, Y. Gesture avatar: A technique for
operating mobile user interfaces using gestures. In 26. Zhai, S., Kristensson, P. O., and Appert, C. Foundational
Proceedings of CHI, ACM (2011), 207–216. Issues in Touch-Surface Stroke Gesture Design. Now
Publishers Inc., Hanover, MA, USA, 2012.

View publication stats

You might also like