2016 15th International Conference on Frontiers in Handwriting Recognition

On the Design of Personal Digital Bodyguards:

Impact of Hardware Resolution on Handwriting Analysis

Daniel Martı́n-Albo Luis A. Leiva Réjean Plamondon

PRHLT Research Center Sciling Laboratoire Scribens
Universitat Politècnica de València Valencia, Spain École Polytechnique de Montréal
Valencia, Spain [email protected] Montréal, Canada
[email protected] [email protected]

Abstract—Handheld touch-capable devices have become one of numerous cognitive tasks leading to the production of
of the most popular and fastest growing consumer products. It words from the motor action plans that have been learned
seems logical therefore to think of such devices as Personal Dig- over the years. According to the Kinematic Theory of
ital Bodyguards (PDBs) in charge for example of biometrical,
biomedical, and neurocognitive monitoring by just inspecting Human Movements [3], [4], these plans activate specific
the user’s handwriting activity. However, it is unclear whether neuromuscular networks to produce a given pen tip trajectory
the hardware of today’s devices is capable to handle this task. by combining lognormal strokes, the fundamental units of
To this end, we conducted a comparative study regarding the handwriting movements [5].
capabilities of past and current tablets to allow for the design of Most of the research regarding this theory has been done
PDBs based on the exploitation of the Kinematic Theory. Our
study shows that, while some improvements are still necessary in well-controlled protocols and experimental setups, using
at the sampling frequency level, the conclusions drawn by the standard digitizers characterized by their stable sampling
Kinematic Theory can be directly transferred to PDBs. frequency and high spatial resolution. One practical ques-
tion that emerges when it comes to making a technology
I. I NTRODUCTION transfer toward handheld devices (e.g. tablets, phablets,
smartphones...) is the following. Is today’s hardware ready
Many human motor control theories rely on the assumption for such a move?
that dynamic and kinematic information is encoded in one In this context, it is far from being certain that the
way or another in the user’s neural activity. Although the conclusions drawn by the Kinematic Theory can be directly
specific underlying processes are still under investigation, transferred to PDBs, at least in the present status of the device
from an engineering and computational perspective, this development ecosystem, where the sampling frequencies are
assumption provides practical guidelines for designing and much smaller than those of the classical digitizers and often
implementing innovative devices and products for different not stable. Furthermore, the spatial resolution is lower and
applications. This is particularly true when it comes to hand- the touchscreen sensitive area might not be homogeneous
writing and gesture-based interaction, for which the number since this is not a requirement in most commonly used tasks
of input devices has increased recently. Ergonomic handheld like e.g. browsing the Web, sending an SMS, or operating a
slate tablets and laptops trapped in tablet bodies, with styli camera application.
or touch gestures replacing computer mice and keyboards, In this paper, we present the results of a comparative study
deliver a natural writing and drawing experience [1]. A regarding the capabilities of past and current tablets to be
similar breakthrough is underway in the smartphone industry. used in the design of PDBs. Our study is mostly exploratory
In the forthcoming years, the ubiquity of smartphones but firmly based on the exploitation of the Kinematic Theory.
and tablets, along with their increased computing power and We show that, while some improvements are necessary at
sensing capabilities, will make it possible to convert these the sampling frequency level, the conclusions drawn by the
devices into Personal Digital Bodyguards (PDBs). Among Kinematic Theory can be directly transferred to PDBs.
other things, PDBs will leverage handwriting activity to
monitor the user’s motor control, being able to detect e.g. II. R ELATED W ORK
stress, aging, and health problems. In this context, PDBs In the past, it has been argued that pen-based digitizers lack
provide an intelligent solution for biometrical, biomedical and sufficient sensitivity [6]. However, this statement no longer
neurocognitive monitoring and in fact are within reach [2]. holds true. For example, Elliott [7] sought to understand
However, the realization of this vision is a difficult chal- how the individual variables of handwriting vary across
lenge. Indeed, handwriting entails complex neuromotor skills. devices. It was found that there are significant differences
Producing a handwritten message requires the performance in signature traits across devices, but these variables are not

2167-6445/16 $31.00 © 2016 IEEE 174

DOI 10.1109/ICFHR.2016.40
significantly different. Today, handwriting accuracy is a still 2
4
matter of concern for tablet users (finger writing behavior),
influenced by many factors including e.g. frequency sampling,
the texture of the screen, or the responsiveness of the device. 1
Many researchers have focused on identifying the activities
that the stylus is most beneficial for. Among other findings,
device and task interactions have been largely confirmed, with
the stylus identified as optimal for compound tasks, crossing
tasks, radial steering, selection, stroke-based gestures, and
shape tracing tasks [8]. Forlines et al. [9] investigated the 6
3
differences between direct touch and mouse input on tabletop
5
displays. They observed that for bimanual tasks performed,
users benefit from direct-touch input. However, mouse input Figure 1. A handwritten letter ‘a’. The handwriting trajectory (green thick
may be more appropriate for a single user working on tabletop line) is described by the temporal overlap of a series of strokes (black
tasks requiring only single-point interaction. In addition, dashed arcs). Each stroke is described by a lognormal equation.
Zabramski et al. [10], [11] compared the performance of
mouse, pen, and touch input in a line-tracing task. It was
observed that touch input was the worst performer in terms The basic idea behind the Kinematic Theory is that the
of accuracy but was the fastest in terms of speed. neuromuscular network involved in the production of a human
Other researchers have focused on the effect of the device movement can be considered as a linear system made up
on the usability and user experience of digital handwriting. of a large number of coupled subsystems [3], [17], [4]. For
For example, Ward and Phillips [12] found several misun- example, when writing on a paper sheet we use from the
derstood performance characteristics of tablet digitizer that shoulder down to the joints of the fingers, each of which
many impact the usability of interactive applications. Also, must be controlled by the muscle groups attached to them.
Annett [8] found that latency, unintended touch, and stylus The resulting velocity profile of a specific neuromuscular
accuracy have a significant impact on the user experience. system converges toward a lognormal function, that is:
Overall, high writing resolution is achieved with higher v (t) = DΛ(t; t0 , μ, σ 2 ) (1)
sampling rates on the capturing device. However, researchers
have shown that some applications do not need such a being
high sampling rates to achieve good results in practice. For −[ln(t−t0 )−μ]2
1
example, Junker et al. [13] found that there is almost no Λ(t; t0 , μ, σ 2 ) = √ e 2σ 2 (2)
loss in accuracy for sampling rates greater than 20 Hz and σ 2π(t − t0 )
resolutions greater than 2 bits. Vatavu [14] analyzed the where D describes the amplitude of the input command;
effect of sampling rate on the performance of template-based t0 is the time occurrence of the input command; μ is the
stroke gesture recognizers. It was found that as few as 6 neuromuscular system time delay and σ is the neuromuscular
sampling points per gesture example are sufficient to attain system response time.
competitive recognition accuracy. There are many models derived from this lognormal
Despite the fundamental research conducted in previous paradigm, among which the Sigma-Lognormal model (ΣΛM)
works, to the best of our knowledge, a systematic examination is the latest and more complete representation [18]. Unlike
of hardware resolution on handwriting analysis is lacking in previous models, the ΣΛM does not assume that the involved
the research literature. Therefore, a study like the one we neuromuscular systems are working in precisely opposite
conducted in this paper is both timely and necessary. directions. The synergy emerging from the interaction and
coupling of many of these neuromuscular systems results in
III. K INEMATIC T HEORY the generation of any complex movements, not limited to a
single stroke.
The Kinematic Theory is aimed at explaining the genera-
According to the ΣΛM, the velocity of a complex move-
tion and control of human movements. This theory has been
ment (Figure 1) is described by the temporal overlap of the
proved in the past years to be one of the best approaches
velocities vi (t) of each involved stroke [19]:
to describe the global properties of the neuromuscular
networks involved in a synergistic action [15], [16]. It N
 N 
 
cos φi (t)
proposes explanations about the emergence of the basic v (t) = vi (t) = Di Λ(t; t0i , μi , σi2 ) (3)
sin φi (t)
kinematic relationships and psychophysical laws that have i=1 i=1

been consistently reported in the studies dealing with human where N represents the number of strokes and φi (t) is
movements [15]. the direction profile for each stroke, described by an error

175
function: Table I
   D EVICE SPECIFICATIONS .
θ ei − θ s i ln(t − t0i ) − μi
φi (t) = θsi + 1 + erf √ (4)
2 σi 2 Manufacturer Device Size Resolution Input
where θsi is the starting direction and θei is the ending Wacom Bamboo 6.1” 2 540 LPI∗ Pen/Touch1
direction of the i-th stroke. Apple iPad mini 7.9” 326 PPI Touch
Lenovo ThinkPad 10.1” 215 PPI Pen/Touch2
Finally, the x(t) and y(t) Cartesian coordinates can be
∗
calculated integrating v (t): 1
1 PPI ≈ 2 LPI
Touch input was disabled.
     N  t 2 Pen input was disabled.
x(t) x
= 0 + vi (τ ) dτ (5)
y(t) y0 t0
i=1 i

or alternatively, x(t) and y(t) can also be computed directly of {x,y,t} tuples. No restriction was placed to the sam-
from the Sigma-Lognormal parameters [20]: pling frequency, thereby obtaining the maximum sampling
     N   frequency possible for each device.1
x(t) x Di sin φi (t) − sin θsi
= 0 +
y(t) y0 θ − θsi − cos φi (t) + cos θsi
i=1 ei
(6) B. Participants
The generation of these complex movements obeys the We recruited 12 participants aged 23–46 (M=28, SD=2.3)
lognormality principle [21]. This principle states that a user using our University’s mailing lists. We intentionally wanted a
in total control of his movements produces the minimum rather broad sample and recruited participants with many dif-
number of perfect lognormal strokes in order to generate ferent backgrounds; e.g. Mechanical Engineering, Computer
the intended hadwriting trajectory. In contrast, when the Science, or Physics. There was no economic compensation for
user is not in full control, the produced strokes will not be the participants, who just provided us with raw handwritten
ideal lognormals or she will use a large number of these data.
to produce the movement. Therefore, the lognormality of
velocity profiles can be interpreted as reflecting the behavior
of users who are ideal motion planners. C. Design and Procedure
In sum, the ΣΛM provides a solid framework to study the We used a repeated measures within-subjects design, i.e.,
generation of complex human movements, as defined by the all participants tested all devices. Each participant had to
control parameters (t0 , D, θ) and the peripheral parameters handwrite ten common English words, extracted from the
(μ, σ) provided by the model. Our evaluation is performed list of “Most common words in English,” according to the
by tapping into this model, as discussed in the next section. Oxford English Dictionary.2 Words were chosen at random
IV. E VALUATION from this list, with the only restriction that they should be
at least three characters long. Each participant entered each
Participants were told to write a series of words on three
word ten times, in order to control for variability, resulting in
different devices. Then, each word was reconstructed accord-
100 samples per participant and device (see Figure 2), 3 600
ing to the ΣΛM. Finally, the performance of each device was
samples in total. We used Latin squares (pseudo-random
assessed through a number of assessment measures derived
condition assignments) to counterbalance the order in which
from the ΣΛM. Below we describe the experimental setup
devices would be tested, and to mitigate possible learning
and the evaluation procedure.
effects between trials.
A. Apparatus 1) ΣΛM Reconstruction Procedure: Given that strokes are
Each participant tested three devices, see Figure 2: a “hidden” inside a handwriting movement, a ΣΛM reconstruc-
Wacom Bamboo Pen & Touch digitizer, an Apple iPad mini 2, tion is needed to perform a “reverse engineering” process,
and a Lenovo ThinkPad 1st gen. We used the first one as in order to uncover the values of the stroke parameters that
a baseline device against which the other devices should best explain the observed velocity profile. For this, we used
be compared, since previous works have used the Wacom the procedure presented in [23]. As discussed later, a good
for ΣΛM analysis [22]. Table I summarizes the technical reconstruction is expected to have the following properties:
specifications of these devices. (1) the reconstruction quality should be higher than a preset
We developed an HTML5 application that rendered a web threshold and (2) for a given reconstruction, the smallest
canvas on which participants could write. The application number of strokes is preferable.
captured both pen and touch coordinates via event listeners,
1 The Wacom digitizer achieved 120 Hz of temporal resolution on average,
together with the timestamp in which the event occurred.
whereas both tablets achieved around 60 Hz.
This way, each word was encoded as an on-line sequence 2 http://www.oxforddictionaries.com

176
 Figure 2. Experimental setup. From left to right: Wacom digitizer, iPad tablet, and ThinkPad tablet.

Wacom iPad ThinkPad

Figure 3. The word “year” written by one of the participants using the different devices.

2) Assessment Measures: We estimate the quality of 4

a reconstruction using three standard measures. The first
Duration (s)

measure is the signal to noise ratio (SNR) between the

original and the reconstructed velocity profile: 2

T
v(t)2
t=1 
SNR = 10 log T
(7)
t=1 v(t) − v (t)2 0
Wacom iPad ThinkPad
where v (t) is the analytic velocity, v(t) is observed velocity Figure 4. Differences in average writing duration (the lower the better).
and T is the duration of the handwriting movement. Previous Error bars denote 95% CIs.
works suggest that different SNR thresholds can be used to
quantify what is considered as a good reconstruction [23],
the Wacom digitizer achieves the highest value, followed by
[24], here 25 dB will be used as upper bound.
the iPad and the ThinkPad. However, with respect to the
The second measure is the number of lognormal strokes,
number of lognormals (nbLog), both the Wacom digitizer
nbLog—or N in Equations (3), (5) and (6)—used in the
and the iPad tablet achieve similar values, and the samples
reconstruction. As previously commented, the smaller this
written on the ThinkPad tablet have 60% more lognormals
number the better.
on average than the other devices. Finally, regarding the
Finally, the third measure evaluates the reconstruction
ratio between SNR and nbLog, the Wacom digitizer achieves
quality according to the lognormality principle, by calculating
the highest value, followed by the iPad (14% less) and the
the ratio between SNR and nbLog. The higher this ratio, the
ThinkPad (32% less).
better. Overall, given two reconstructions with the same
number of strokes, a bigger SNR is always preferable. And
if both reconstructions achieve the same SNR, a smaller
20
number of lognormal strokes is desirable.
SNR (dB)

D. Results 10
First, we measured the average writing duration per device.
As shown in Figure 4, writing on the handheld devices is
0
slower. In particular, it is half slower on the iPad and two Wacom iPad ThinkPad
times slower on the ThinkPad. Figure 5. Differences in SNR (the higher the better). Error bars denote
Next, we looked at the ΣΛM reconstruction quality 95% CIs.
measures; see Figures 5 to 7. Unsurprisingly, in terms of SNR,

177
 which is considered to be appropriate for human movement
40
analysis [25].
# lognormals

On the other hand, we observed that participants wrote

much more carefully on the ThinkPad tablet to avoid
20
handwritten words looking “wobbly” due to the low screen
resolution. This fact may explain to a great extent the signif-
icant higher number of lognormals required to reconstruct
0
Wacom iPad ThinkPad each handwritten trajectory on average (see Figure 6).
Figure 6. Differences in the number of lognormals (the lower the better). However, even when the writing duration is higher than
Error bars denote 95% CIs. usual (see Figure 4), it can be observed that all devices
achieved a comparative performance in terms of SNR/nbLog
0.8 (see Figure 7). The differences among these ratios suggest
SNR / nbLog

that high-quality reconstruction is still achievable in finger

0.6
writing. This result is especially important because it confirms
0.4 previous works’ findings that showed that the Kinematic
Theory was not limited to pen-based handwriting, but could
0.2
be used in a more general context. For example, reproducing
0 wrist movement and eye saccades [4], 2D and 3D arm
Wacom iPad ThinkPad movements [26], and more recently, stroke gestures [24].
Figure 7. Differences in SNR/nbLog (the higher the better). Error bars It must be pointed out that the use of a stylus greatly
denote 95% CIs.
improves performance, as compared to finger writing. Also,
the friction is one of the major factors in determining
To analyze better the differences between these devices we the accuracy of handwritten text, but unfortunately modern
performed an Analysis of Variance (ANOVA) for each quality tablets like the ones we used in our study do not allow to
measure. The ANOVA test revealed statistical significance control this factor. However, despite of these observations,
in all cases excepting the ratio between SNR and nbLog: our findings open up the possibility of using the Kinematic
writing duration [F2,33 = 9.76, p < .001, ηp2 = 0.37], Theory in numerous applications like games or gesture-
SNR [F2,33 = 3.94, p = .029, ηp2 = 0.19], nbLog based applications. For biomedical applications, one question
[F2,33 = 3.47, p = .042, ηp2 = 0.17], and SNR/nbLog remains to be investigated and in fact this will be the topic of
[F2,33 = 3.03, p = .062, ηp2 = 0.15]. Effect sizes suggest a follow up study. Do the ΣΛM parameters extracted from
small to moderate practical significance of the results. the trajectories differ greatly form one device to another? In
Therefore, we conducted post-hoc pairwise t-tests (two preliminary experiments we have observed that the peripheral
tails, Bonferroni-Holm corrected for multiple comparisons) parameters—μ and σ in Equations (1) to (4)—do not vary
where the ANOVA test reported statistical significance. The too much from device to device for the same participant.
post-hoc tests provide specific information on which devices For example, the user in Figure 2 has μWacom = −1.52,
perform significantly different from each other. μiPad = −1.58, μThinkPad = −1.80, and σ Wacom = 0.3,
In terms of writing duration, the differences between the σ iPad = 0.28, σ ThinkPad = 0.28. However, these experiments
Wacom digitizer and the ThinkPad tablet are statistically are outside the scope of this paper and will be the subject
significant (p < .001). Thus, the ThinkPad tablet performs of study in future work. Eventually, the answer to such a
significantly worse than the Wacom digitizer in this regard. question will be particularly determinant when it comes to
In terms of SNR, the differences between the ThinkPad track the neuromotricity of a person from different types
and iPad tablets are not statistically significant (p > .05). of devices. We also plan to test more devices that cover
However, the Wacom digitizer performs significantly better different variabilities in e.g. size, resolution, and sampling
than both tablets in this regard (p = .027). Finally, in terms of frequency. All in all, this work should be seen as one corner
nbLog, the differences between all devices are not statistically stone of a broad series of potential applications of the PBS
significant (p > .05), suggesting that all devices achieve concept.
equally similar performance in this regard. Other comparisons
were not found to be statistically significant. VI. C ONCLUSION

V. D ISCUSSION AND F UTURE W ORK This study is the very first to address fundamental questions
regarding the interoperability of handheld devices based on
As it can be seen from the above results, all of the studied
the exploitation of the Kinematic Theory. Previous research
devices could be used in the context of the Kinematic Theory
in the context of this theory has been done in well-controlled
since, among other findings, the handwritten trajectories can
protocols and experimental setups, using standard digitizers
be reproduced with SNR higher than 20 dB (see Figure 5),

178
characterized by their stable and high sampling frequency [12] J. R. Ward and M. J. Phillips, “Digitizer technology: Perfor-
and spatial resolution. mance characteristics and the effects on the user interface,”
So far, we can anticipate that improvements will be IEEE Computer Graphics and Applications, vol. 7, no. 4, pp.
31–44, 1987.
necessary at the sampling frequency level. However, our
study has shown that today’s hardware is ready to make a [13] H. Junker, P. Lukowicz, and G. Troster, “Sampling frequency,
technology transfer toward handheld devices. With this, the signal resolution and the accuracy of wearable context recog-
PDB concept and ideas become finally realizable. Looking nition systems,” in International Symposium on Wearable
forward, we believe this paper will inform researchers and Computers (ISWC), 2004, pp. 176–177.
practitioners about the design of PDBs. Now we can be [14] R.-D. Vatavu, “The effect of sampling rate on the performance
confident that it is possible to derive practical guidelines of template-based gesture recognizers,” in Proc. Intl. Conf. on
for implementing innovative devices in charge of active Multimodal Interaction (ICMI), 2011, pp. 271–278.
neurocognitive monitoring.
[15] R. Plamondon and A. Alimi, “Speed/accuracy tradeoffs in
ACKNOWLEDGMENTS target directed movements,” Behav. Brain Sci., vol. 20, no. 2,
This work has been partially supported by the EC’s H2020 pp. 279–349, 1997.
program through grant 674943 (READ project) and by the [16] W. Guerfali and R. Plamondon, “A new method for the
NSERC through grant RGPIN-2015-06409. analysis of simple and complex planar rapid movements,”
J. Neuroscience Methods, vol. 82, no. 1, 1998.
R EFERENCES
[1] F. Matulic and M. Norrie, “Empirical evaluation of uni- [17] C. Ghez and J. Krakauer, “Voluntary movement,” Principles
and bimodal pen and touch interaction properties on digital of neural science, vol. 3, pp. 622–624, 1991.
tabletops,” in Proc. Intl. Conf. on Interactive Tabletops and
Surfaces (ITS), 2012, pp. 143–152. [18] R. Plamondon and M. Djioua, “A multi-level representation
paradigm for handwriting stroke generation,” Hum. Mov. Sci.,
[2] R. Plamondon, “Personal digital bodyguards for e-security, vol. 25, no. 4–5, pp. 586–607, 2006.
e-health and e-learning,” in Proc. Biennial Conf. of the Int.
Graphonomics Society, 2015, keynote opening lecture. [19] R. Plamondon and W. Guerfali, “The 2/3 power law: When
and why?” Acta psychologica, vol. 100, no. 1, pp. 85–96,
[3] ——, “A kinematic theory of rapid human movements. Part I: 1998.
Movement representation and control,” Biol. Cybern., vol. 72,
no. 4, pp. 295–307, 1995. [20] C. O’Reilly and R. Plamondon, “Development of a Sigma-
Lognormal representation for on-line signatures,” Pattern
[4] ——, “A kinematic theory of rapid human movements. Part Recogn., vol. 42, no. 12, pp. 3324–3337, 2009.
II: Movement time and control,” Biol. Cybern., vol. 72, no. 4,
pp. 309–320, 1995. [21] R. Plamondon, C. O’Reilly, C. Rémi, and T. Duval, “The
[5] A. Woch and R. Plamondon, “Using the framework of the lognormal handwriter: learning, performing, and declining,”
kinematic theory for the definition of a movement primitive,” Front. Psychol., vol. 4, no. 1, pp. 945:1–945:14, 2013.
Motor Control, vol. 8, no. 4, pp. 547–557, 2004.
[22] C. O’Reilly and R. Plamondon, “Can computer mice be used as
[6] R. J. Elble, R. Sinha, and C. Higgins, “Quantification of tremor low-cost devices for the acquisition of planar human movement
with a digitizing tablet,” Journal of Neuroscience Methods, velocity signals?” Behav. Res. Methods, vol. 43, no. 1, pp.
vol. 32, no. 3, pp. 193–198, 1990. 229–238, 2011.

[7] S. J. Elliott, “Differentiation of signature traits vis-à-vis mobile- [23] D. Martı́n-Albo, R. Plamondon, and E. Vidal, “Improving
and table- based digitizers,” ETRI Journal, vol. 26, no. 6, pp. sigma-lognormal parameter extraction,” in Proc. Intl. Conf. on
641–646, 2004. Document Analysis and Recognition (ICDAR), 2015.

[8] M. K. Annett, “The fundamental issues of pen-based inter- [24] L. A. Leiva, D. Martı́n-Albo, and R. Plamondon, “Gestures à
action with tablet devices,” Ph.D. dissertation, University of go go: Authoring synthetic human-like stroke gestures using
Alberta, 2014. the kinematic theory of rapid movements,” ACM T. Intel. Syst.
Tec., vol. 7, no. 2, 2015.
[9] C. Forlines, D. Wigdor, C. Shen, and R. Balakrishnan, “Direct-
touch vs. mouse input for tabletop displays,” in Proc. SIGCHI [25] A. Fischer, R. Plamondon, C. O’Reilly, and Y. Savaria,
Conf. on Human Factors in Computing Systems (CHI), 2007, “Neuromuscular representation and synthetic generation of
pp. 647–656. handwritten whiteboard notes,” in Proc. Intl. Conf. on Frontiers
in Handwriting Recognition (ICFHR), 2014, pp. 222–227.
[10] S. Zabramski, “Careless touch: A comparative evaluation of
mouse, pen, and touch input in shape tracing task,” in Proc. [26] N. Leduc and R. Plamondon, “A new approach to study human
Australian Computer-Human Interaction Conference (OzCHI), movements: The three dimensional Delta-Lognormal model,”
2011, pp. 329–332. in Proc. Biennial Conf. of the Intl. Graphonomics Society
[11] S. Zabramski and W. Stuerzlinger, “The effect of shape (IGS), 2001, pp. 98–102.
properties on ad-hoc shape replication with mouse, pen, and
touch input,” in Proceeding of the 16th International Academic
MindTrek Conference, 2012, pp. 275–278.

179