CHAPTER-1

INTRODUCTION

1.1 INTRODUCTION OF ANDROID

Android is a Linux-based operating system for mobile devices such as smart phones and tablet
computers. It is developed by the Open Handset Alliance led by Google. Google purchased the
initial developer of the software, Android Inc., in 2005.The unveiling of the Android
distribution in 2007 was announced with the founding of the Open Handset Alliance, a
consortium of 86 hardware, software, and telecommunication companies devoted to advancing
open standards for mobile devices. Google releases the Android code as open-source, under the
Apache License. The (AOSP) is tasked with the maintenance and further development of
Android .Android has a large community of developers writing applications ("apps") that extend
the functionality of the devices. Developers write primarily in a customized version of Java.
Apps can be downloaded from third-party sites or through online stores such as Google Play
(formerly Android Market), the app store run by Google.

Fig 1.1-Android symbol

Touch screens are a great way to interact with applications on mobile devices. With a touch
screen, users can easily tap, drag, fling, or slide to quickly perform actions in their favourite
applications. For app developers. The Android framework makes it's easy to recognize simple
actions, like a swipe, but it has been more difficult to handle complicated gestures, sometimes

1
requiring developers to write a lot of code. Android 1.6 and later provide gesture recognition
support and lets you store, load, draw, and recognize gestures. Android 1.6 and higher SDK
platforms include a new application pre-installed on the emulator, called Gestures Builder. We
can use this application to create a set of pre-defined gestures for your own application. Every
time we add or edit a gesture in the Gestures Builder, a file is generated on the emulator's SD
card.

We have a set of pre-defined gestures and we have to load it inside our application. This is
done using a Gesture Libraries class .To start recognizing gestures in our application; we have to
add a Gesture Overlay View to the XML layout. A gesture overlay acts as a simple drawing
board on which the user can draw his gestures. When you write something on the gesture
overlay, the Gesture Library try to recognize the gesture. In return, you will get a list of
Prediction instances, each with a name - the same name you entered in the Gestures Builder and
a score.

1.2 INTRODUCTION OF GESTURE RECOGNITION

Gesture recognition is a topic in computer science and language technology with the goal of
interpreting human gestures via mathematical algorithms. Gestures can originate from any
bodily motion or state but commonly originate from the face or hand. Current focuses in the
field include emotion recognition from the face and hand gesture recognition. Many approaches
have been made using cameras and computer vision algorithms to interpret sign language.

Fig-1.2-Various gestures demonstrated

2
However, the identification and recognition of posture, gait, proxemics, and human behaviors is
also the subject of gesture recognition techniques. Gesture recognition can be seen as a way for
computers to begin to understand human body language, thus building a richer bridge between
machines & humans than primitive text user interfaces or even GUIs, which still limit the
majority of input to keyboard and mouse.

Gesture recognition enables humans to interface with the machine (HMI) and interact naturally
without any mechanical devices. Using the concept of gesture recognition, it is possible to point
a finger at the computer screen so that the cursor will move accordingly. This could potentially
make conventional input devices such as mouse, keyboards and even touch-screens redundant.

Gesture recognition can be conducted with techniques from computer vision and image
processing. Interface with computers using gestures of the human body, typically hand
movements.

Fig 1.3-Gestures with functions

In gesture recognition technology, a camera reads the movements of the human body and
communicates the data to a computer that uses the gestures as input to control devices or
applications. For example, a person clapping his hands together in front of a camera can produce
the sound of cymbals being crashed together when the gesture is fed through a computer

3
One way gesture recognition is being used is to help the physically impaired to interact with
computers, such as interpreting sign language. The technology also has the potential to change
the way users interact with computers by eliminating input devices such as joysticks, mice and
keyboards and allowing the unencumbered body to give signals to the computer through
gestures such as finger pointing. Unlike haptic interfaces, gesture recognition does not require
the user to wear any special equipment or attach any devices to the body. The gestures of the
body are read by a camera instead of sensors attached to a device such as a data glove. In
addition to hand and body movement, gesture recognition technology also can be used to read
facial and speech expression (i.e., lip reading), and eye

movements.

Fig-1.4-Performing Gesture

1.3 FEASIBILITY STUDY

Feasibility study is a test of a system proposal according to its workability, impact on the
organization, ability to meet user needs and effective use of resources .Three key considerations
are involved in the feasibility analysis: economic, technical, behavioural. Before entering in to

4
the procedure of system designing, we are obliged to study about the feasibility of introducing a
new computerized system. To replace the existing system with the new one is quite easy, but we
have to vanish the drawbacks of current system and to make the user able to enjoy the
advantages of coming system. The proposed system must be evaluated from a technical,
operational and economic feasibility of developing a computer system. The objective of the
feasibility study is not only to solve the problem but also to acquire the sets of scope. During the
study, the problem is defined is crystallised and aspect of the problem to be included in the
system are determined .Consequently, costs and benefits are estimated with greater accuracy at
this stage.

1.3.1 EXISTING SYSTEM

The need for a gesture recognition system (GRS) has been very prevalent over the past decade.
Many different problems have found their solutions in gesture recognition. More so, many
unique applications have arisen from this technology. Previously, it was inconvenient to interact
with the system during presentation and seminars.

The need for a perceptual human computer interface has been around for sometime now. With
the onset of computer encounters and interactions becoming more prevalent in our daily life, the
need for an intuitive interface to interact with a computer has increased.

1.3.2 LIMITATIONS OF EXISTING SYSTEM

The existing system has got several limitations. Gesture recognition system that recognize
English alphabets, digits & symbols are already available. The gesture given as input on gesture
overlay view is matched with the existing gestures in the gesture library and the matched
alphabet, digit or symbols are displayed in the output screen. It is difficult to incorporate
Malayalam language as it contains vast number of letters and conjuncts that are unique.

5
1.3.3 PROPOSED SYSTEM

To solve the problem in existing system, a system is provided called gesture recognition
system. A primary goal of gesture recognition system is to create a system which can identify
specific human gestures and use them to convey information or for device control . It should
provide the facility to enter Malayalam gestures as inputs and get the corresponding alphabet as
output. Along with that it should process the recognition of English alphabets letters and digits.
Maximum possible gestures of different persons should be identified.

1.3.4 ADVANTAGES OF THE PROPOSED SYSTEM

- Can recognise Malayalam gestures which is not yet developed

- Touch screens are a great way to interact with applications on mobile devices
- Common people can handle
- The Android framework makes it's easy to recognize simple action
- High level of user interaction-building a richer bridge between machines and humans
- Touch screen-based interfaces be usable by people with all abilities, including blind and
visually impaired persons.

6 6
 CHAPTER-2
SYSTEM DESIGN AND ANALYSIS
Design is the first phase in the development for any engineers project .It is a creative
process. A good design is a key to effective system. From a project management point of
view, software design is conducted in two steps .Preliminary design is concerned with the
transformation of requirements in to data and software architecture. Detailed design focuses
on the refinement to the architectural representation that lead to detailed algorithm data
structure and representation of software.

2.1 DETAILED DESIGN

The design part mainly includes the designing of layouts and a user-defined gesture set.
The gesture recognition service can recognize new performed gestures and provide the
recognition result to android applications (activities) that subscribe to this service. The result
contains the tag of the best matching gesture and the distance of the classified gesture to the
best matching gesture. The smaller the distance the better the similarity. The service supports
different training sets that allow to personalize recorded gestures or to define specific gesture
training sets for individual applications. For gesture recognition the service uses the genetic
algorithm - dynamic time warp which has been extended to classify multidimensional
signals .The design includes,

2.1.1 CREATING THE GESTURE LIBRARY

Android 1.6 and higher SDK platforms include a new application pre-installed on the
emulator, called GestureBuilder. We can use this application to create a set of pre-defined
gestures for our own application. It also serves as an example of how to let the user define his
own gestures in our applications. We can find the source code of Gestures Builders in the
samples directory of each SDK platform. In our project we will use Gestures Builder to
generate a set of gestures for us. A gesture is always associated with a name. That name is
very important because it identifies each gesture within our application. The names do not
have to be unique.

7
 Fig.2.1-Gesture Builder

Actually it can be very useful to have several gestures with the same name to increase the
precision of the recognition. Every time we add or edit a gesture in the Gestures Builder, a
file is generated on the emulator's SD card, /SD card/gestures. This file contains the
description of all the gestures, and you will need to package it inside your application inside
the resources directory, in/res/raw.

Fig-2.2-Gesture Builder

8
 Fig-2.3-Gesture Builder

2.1.2 LOADING THE GESTURE LIBRARY

We have a set of pre-defined gestures, we must load it inside our application. This
can be achieved in several ways but the easiest is to use the GestureLibraries class.
Hierarchical representation of library is shown in figure 3.1.2. We can easily load libraries
from other sources, like the SD card, which is very important if we want your application to
be able to save the library; a library loaded from a raw resource is read-only and cannot be
modified.

9
 Fig-2.4-Creating Gesture

Fig 2.5-Creating a gesture

10
2.1.3 RECOGNIZING GESTURES

To start recognizing gestures in our application, we have to add a GestureOverlayView to our

XML layout .We are creating layouts in XML by just make use of drag and drop
mechanism .The layouts include layout for gesture builder, gesture library, Renaming
window and also input. Gesture overlay view i.e. the input window which acts as a simple
drawing board on which the user can draw his gestures. We can tweak several visual
properties, like the colour and the width of the stroke used to draw gestures, and register
various listeners to follow what the user is doing. The most commonly used listener is,
GestureOverlayView.OnGesturePerformedListener, which fires whenever a user is done
drawing a gesture .The input window contains a button for selecting the languages English,
Malayalam ,or digits, toggle buttons for uppercase ,lowercase, or caps lock and also for a
button to go to the gesture builder to add new gestures.

2.1.4 TYPES OF GESTURES

Include maximum possible gestures in the gesture library so that it will be able to
recognize gestures drawn by people having different handwriting. One of the major
applications is, it should be useful to the peoples having poor eyesight. Until recently, most
touch screens provided few or no accessibility features, leaving them largely unusable by
blind people .So during design we also wished to provide gestures in our application that will
be appropriate for a blind user. Although blind people may use the same hardware as their
sighted peers, it is possible that they will prefer to use different gestures, or that they will
perform the same gestures differently than a sighted person. Sighted people perform gestures
differently when they lack visual feedback, and it is reasonable to assume that a blind person
may also perform gestures differently than a sighted person.

11
 Fig 2.6- (Two representative versions of a triangle gesture produced by a blind person (left)
and a sighted person (right).

One speciality is that in the gesture library can include shorthand method to specify certain
words that we are constantly using. This reduces the difficulty in typing messages. As we can
include Malayalam gestures we will be able to send messages in our mother tongue. In this
way inputting in Malayalam becomes easy.

Fig-2.7-Different type of gestures

Fig-2.8-Different types of gestures

12
 Fig-2.9-Different types of gestures

2.2 FRAMEWORK OVERVIEW

The framework is build using Java language. The choice to use Java was to achieve a higher
number of devices with the same framework, allowing its usage by any accelerated/touch
device that can handle the acceleration data and/or touch data, and that has a Java virtual
machine, like many devices from different manufacturers: Blackberries, Nokia, Sony Ericson,
Motorola, Android, LG and Samsung. Also this framework can be used in a PC with a touch
device like Microsoft surface or an accelerometer device like the Wiimote. Gesture
recognition with touch/accelerated devices are represented by their patterns of the input data.
The recognition is made by comparing the input pattern with the database pattern, checking if
they match. In order to extract the pattern from the input data stream and comparing with the
database pattern, this data must be prepared and analysed. This work uses Hidden Markov
Models in order to fulfil that need. In order to build a database, the framework needs to train
and saves a set of gestures, which are also used by the mobile phone for recognition.

Fig-2.10-Block Diagram Of Gesture Recognition Technology

13
2.2.1 Sensor Processing

Sensor processing is the neurological process that organizes sensation from one’s own body
and the environment, thus making it possible to use the body effectively within the
environment. Specifically, it deals with how the brain processes multiple sensory modality
inputs, such as proprioception, vision, auditory system, tactile, olfactory, vestibular
system, interoception, and taste into usable functional outputs.

It has been believed for some time that inputs from different sensory organs are processed in
different areas in the brain. The communication within and among these specialized areas of
the brain is known as functional integration. Newer research has shown that these different
regions of the brain may not be solely responsible for only one sensory modality, but could
use multiple inputs to perceive what the body senses about its environment. Multisensory
integration is necessary for almost every activity that we perform because the combination of
multiple sensory inputs is essential for us to comprehend our surroundings.

2.2.2 Feature Extraction

In machine learning, pattern recognition and in image processing, feature extraction starts
from an initial set of measured data and builds derived values (features) intended to be
informative, non redundant, facilitating the subsequent learning and generalization steps, in
some cases leading to better human interpretations. Feature extraction is related
to dimensionality reduction.

When the input data to an algorithm is too large to be processed and it is suspected to be
redundant (e.g. the same measurement in both feet and meters, or the repetitiveness of images
presented as pixels), then it can be transformed into a reduced set of features (also named
features vector). This process is called feature extraction. The extracted features are expected
to contain the relevant information from the input data, so that the desired task can be
performed by using this reduced representation instead of the complete initial data.

2.2.3 Gesture Classification

The classifier is used in order to identify gesture selecting the gesture with more like hood
between the input gesture and the database gesture. This work uses a naive Bayes classifier
also called simple Bayesian. Naive Bayes is simple probabilistic classifier based on the so-

14
called Bayesian theorem and it is a well-known algorithm both in statistics and machine
learning .There will be some probability value like prediction size, so that any gesture
probability below that threshold is considered as a noise or poor match. Here if the prediction
value is less than 1 it is taken as a poor match.

The proposed framework has the following steps:

- Segmentation: is used to determine when the gesture begin sand when it ends
- Filtering: is used in order to eliminate some parts of the data stream that do not contribute to
the gesture;
- Quantitizer: is used to approximate the stream of input data into a smaller set of values;
- Model: is used to compute likelihood of analysed gestures.
- Classifier: is used in order to identify the input gesture accordingly to the database
It is possible to notice that the framework has two distinct modes: one for gesture training,
i.e., build the database, and one for gesture recognition, i.e., comparing the input gesture with
the database.

2.2.1 SEGMENTATION

Segmentation is used mainly to automatically determinate the begin and end of the
gesture. This identification in touch gestures is very easy since the begin of the gesture is
when the user first touch the screen and it ends when the user release the finger/pen from the
screen. In order to correct segment ate an accelerometer gesture, a definition of this kind of
gesture is needed. This definition is made during the observation of the accelerated data and
the movement of user during the recognition of different gestures. Normally, gestures begin
with a fast acceleration, a continuous direction change during the gesture, and it ends with a
stop of the movement.

2.2.2 FILTERING

This pass is used in order to eliminate some parts of the data stream that do not contribute to
the gesture. This work uses two kinds of filters in order to eliminate noises and data that are
very similar. When a gesture is made, the data stream of the gesture may contain errors that if
are sent to the HMM, some errors on the recognition may occur. In order to avoid such errors,
a low pass filter is applied which is a very common filter used for noise remove. When the
gesture is made, there are a lot of data on the data stream that does not contribute to the

15
overall characteristic of the gesture. In order to diminished the data passed to the HMM, this
work uses an idle threshold filter.

2.2.3 QUANTITIZER

This step is only used for accelerated gestures. Because the accelerometer continues sends its
data to the processor, the amount of data may be to big to for handling into a single HMM.
Also, since the amount of RAM memory of mobile phones is limited, the use of a quantitizer
keeps less information in the gesture database. This work uses a k-mean algorithm which is a
method of cluster analyser. The algorithm aims to partition n observations into k clusters in
which each observation belongs to the cluster with the nearest mean.

2.2.4 CLASSIFIER

The classifier is used in order to identify gesture selecting the gesture with more like hood
between the input gesture and the database gesture. This work uses a naive Bayes classifier
also called simple Bayesian. Naive Bayes is simple probabilistic classifier based on the so-
called Bayesian theorem and it is a well-known algorithm both in statistics and machine
learning .There will be some probability value like prediction size, so that any gesture
probability below that threshold is considered as a noise or poor match. Here if the prediction
value is less than 1 it is taken as a poor match.

16
 Fig-2.11- Diagram of how a classifier works

2.2.5 GESTURE RECORDER

The gesture recorder component listens to events that are emitted from the accelerometer
sensors, so it steadily monitors the acceleration of the device. A gesture is detected if the
absolute value of the acceleration exceeds a special threshold for a sufficient period of time.
Thus, it is assured that noises produced by unintended movements of the hand do not emit
gestures.

Fig-2.12-Gesture Recorder

2.2.6 GESTURE RECOGNITION SERVICE

The Gesture Recognition Service is a service that can be subscribed by any android
application. Gesture recognition is running in the background and reacts on new gestures

17
recorded by the Gesture Recorder. Depended on the current mode, that can be set via the
service's interface; new gestures are added to a training set or classified. The result is
delivered to the application that is registered as a listener.

Fig 2.13-Gesture Overlay View

2.2.7 GESTURE TRAINING APPLICATION

The gesture training application is an application that uses the Gesture Recognition Service to
manage and to extend training sets with new gestures. It uses the service interface to create
new training sets, delete training sets, record new gestures, and delete existing gestures.
Furthermore it supports two modes. The first mode is the training mode. When it is active,
new recorded gestures are added to the actual selected training set. Second is the
classification mode to classify the new gestures.

18
 Fig-2.14-Gesture Training Application

2.3 DATAFLOW DIAGRAM

A Dataflow diagram is a graphical technique that depicts information flow and

transforms that are applied as data move from input to output. The DFD is used to represent
increasing information flow and functional details .A Level 0 DFD is called a fundamental
system model or a context model, represents the entire software elements as a single bubble
with input and output indicated by incoming and outgoing arrows respectively .Additional
process and information flow are represented in the next level i.e. Level 1 DFD .Each of the
process represented at level 1 as sub functions of overall system depicted in the context
model.

2.3.1 CONTEXT DIAGRAM

A System Context Diagram (SCD) in software engineering and systems engineering

is a diagram that represents the actors outside a system that could interact with that system.
This diagram is the highest level view of a system. It is similar to a Block diagram.

2.3.2 GESTURE TRAINER ARCHITECTURE

 Fig 2.15

2.4 TECHNIQUES USED

19
2.4.1 DYNAMIC TIME WARPING

In time series analysis, dynamic time warping (DTW) is an algorithm for measuring
similarity between two temporal sequences which may vary in time or speed. For instance,
similarities in walking patterns could be detected using DTW, even if one person was
walking faster than the other, or if there were accelerations and decelerations during the
course of an observation. DTW has been applied to temporal sequences of video, audio, and
graphics data — indeed, any data which can be turned into a linear sequence can be analyzed
with DTW. A well known application has been automatic speech recognition, to cope with
different speaking speeds. Other applications include speaker recognition and
online signature recognition. Also it is seen that it can be used in partial shape
matching application.

In general, DTW is a method that calculates an optimal match between two given sequences
(e.g. time series) with certain restrictions. The sequences are "warped" non-linearly in the
time dimension to determine a measure of their similarity independent of certain non-linear
variations in the time dimension. This sequence alignment method is often used in time series
classification. Although DTW measures a distance-like quantity between two given
sequences, it doesn't guarantee the triangle inequality to hold.

2.4.2 HIDDEN MARKOV MODEL

A hidden Markov model (HMM) is a statistical Markov model in which the system being
modeled is assumed to be a Markov process with unobserved (hidden) states. An HMM can
be presented as the simplest dynamic Bayesian network. The mathematics behind the HMM
was developed by L. E. Baum and coworkers. It is closely related to an earlier work on the
optimal nonlinear filtering problem by Ruslan L. Stratonovich, who was the first to describe
the forward-backward procedure.

In simpler Markov models (like a Markov chain), the state is directly visible to the observer,
and therefore the state transition probabilities are the only parameters. In a hidden Markov
model, the state is not directly visible, but output, dependent on the state, is visible. Each
state has a probability distribution over the possible output tokens. Therefore the sequence of
tokens generated by an HMM gives some information about the sequence of states. Note that

20
the adjective 'hidden' refers to the state sequence through which the model passes, not to the
parameters of the model; the model is still referred to as a 'hidden' Markov model even if
these parameters are known exactly.

Hidden Markov models are especially known for their application in temporal pattern
recognition such as speech, handwriting, gesture recognition, part-of-speech tagging, musical
score following, partial discharges and bioinformatics.

A hidden Markov model can be considered a generalization of a mixture model where the
hidden variables (or latent variables), which control the mixture component to be selected for
each observation, are related through a Markov process rather than independent of each other.
Recently, hidden Markov models have been generalized to pairwise Markov models and
triplet Markov models which allow consideration of more complex data structures and the
modelling of non-stationary data

2.4.3 NAÏVE BAYES

A Bayesian network, Bayes network, belief network, Bayes(ian) model or probabilistic

directed acyclic graphical model is a probabilistic graphical model (a type of statistical
model) that represents a set of random variables and their conditional dependencies via a
directed acyclic graph (DAG). For example, a Bayesian network could represent the
probabilistic relationships between diseases and symptoms. Given symptoms, the network
can be used to compute the probabilities of the presence of various diseases.Formally,
Bayesian networks are DAGs whose nodes represent random variables in the Bayesian sense:
they may be observable quantities, latent variables, unknown parameters or hypotheses.
Edges represent conditional dependencies; nodes that are not connected represent variables
that are conditionally independent of each other. Each node is associated with a probability
function that takes, as input, a particular set of values for the node's parent variables, and
gives (as output) the probability (or probability distribution, if applicable) of the variable
represented by the node. For example, if parent nodes represent Boolean variables then
the probability function could be represented by a table of entries, one entry for each of
the possible combinations of its parents being true or false. Similar ideas may be applied
to undirected, and possibly cyclic, graphs; such are called Markov networks.

21
 CHAPTER-3

APPLICATIONS & ADVANTAGES

3.1 APPLICATIONS

Gesture recognition is useful for processing information from humans which is not conveyed
through speech or type. As well, there are various types of gestures which can be identified
by computers.

• Sign language recognition

Just as speech recognition can transcribe speech to text, certain types of gesture recognition
software can transcribe the symbols represented through sign language into text.

• For socially assistive robotics

By using proper sensors (accelerometers and gyros) worn on the body of a patient and by
reading the values from those sensors, robots can assist in patient rehabilitation. The best
example can be stroke rehabilitation.

• Directional indication through pointing

Pointing has a very specific purpose in our society, to reference an object or location based
on its position relative to ourselves. The use of gesture recognition to determine where a
person is pointing is useful for identifying the context of statements or instructions. This
application is of particular interest in the field of robotics.

• Control through facial gestures

Controlling a computer through facial gestures is a useful application of gesture recognition

for users who may not physically be able to use a mouse or keyboard. Eye tracking in may be
of use for controlling cursor motion or focusing on elements of a display.

• Alternative computer interfaces

Foregoing the traditional keyboard and mouse setup to interact with a computer, strong

gesture recognition could allow users to accomplish frequent or common tasks using hand

22
or face gestures to a camera.

• Immersive game technology

Gestures can be used to control interactions within video games to try and make the game
player's experience more interactive or immersive.

• Virtual controllers

For systems where the act of finding or acquiring a physical controller could require too
much time, gestures can be used as an alternative control mechanism. Controlling secondary
devices in a car, or controlling a television set are examples of such usage.

Fig 3.1-Using Gesture Recognition technology with the help of sensors

• Effective computing

In Effective computing, gesture recognition is used in the process of identifying emotional

expression through computer systems.

• Remote control

Through the use of gesture recognition, "remote control with the wave of a hand" of various
devices is possible. The signal must not only indicate the desired response, but also which
device to be controlled. The ability to track a person's movements and determine what
gestures they may be performing can be achieved through various tools. Although there is a
large amount of research done in image/video based gesture recognition, there is some
variation within the tools and environments used between implementations.

• Depth-aware cameras

23
Using specialized cameras such as time-of-flight cameras, one can generate a depth map of
what is being seen through the camera at a short range, and use this data to approximate a 3d
representation of what is being seen. These can be effective for detection of hand gestures
due to their short range capabilities.

• Stereo cameras

Using two cameras whose relations to one another are known, a 3d representation can be
approximated by the output of the cameras. To get the cameras' relations, one can use a
positioning reference such as a lexian-stripe or infrared emitters. In combination with direct
motion measurement (6D-Vision) gestures can directly be detected.

• Controller-based gestures

These controllers act as an extension of the body so that when gestures are performed, some
of their motion can be conveniently captured by software. Mouse gestures are one such
example, where the motion of the mouse is correlated to a symbol being drawn by a person's
hand, as is the WiiRemote, which can study changes in acceleration over time to represent
gestures.

• Single camera

A normal camera can be used for gesture recognition where the resources/environment
would not be convenient for other forms of image-based recognition. Although not
necessarily as effective as stereo or depth aware cameras, using a single camera allows a
greater possibility of accessibility to a wider audience.

-Answer an incoming call (speaker-ON) with a wave of the hand while driving.

Fig-3.2-Gestures for different functions

24
-Skip tracks or control volume on your media player using simple hand motions- lean back,
and with no need to shift to the device- control what you watch/ listen to.
-Scroll Web Pages, or within an eBook with simple left and right hand gestures, this is ideal
when touching the device is a barrier such as wet hands are wet, with gloves, dirty etc. (we
all are familiar with the annoying smudges on the screen from touching).

Fig-3.3-Gestures for different operations

-Another interesting use case is when using the smartphone as a media hub, a user can dock
the device to the TV and watch content from the device- while controlling the content in a
touch-free manner from afar.

Fig-3.4-Gesture technology used during medical treatment

25
 Fig- 3.5-Zooming in and out

Fig- 3.6-Switching channels by waving a hand

26
 Fig 3.7- Practical advantage

Fig 3.8- Automated Homes

3.2 ADVANTAGES
Gesture recognition, one key example of these sensor-enabled technologies, is achieving
rapid market adoption as it evolves and matures. Although various gesture implementations
exist in the market, a notable percentage of them are based on embedded vision algorithms

27
that use cameras to detect and interpret finger, hand and body movements. Gestures have
been part of humans’ native interaction language for eons. Adding support for various types
of gestures to electronic devices enables using our natural "language" to operate these
devices, which is much more intuitive and effortless when compared to touching a screen,
manipulating a mouse or remote control, tweaking a knob, or pressing a switch.

Gesture controls will notably contribute to easing our interaction with devices, reducing (and
in some cases replacing) the need for a mouse, keys, a remote control, or buttons. When
combined with other advanced user interface technologies such as voice commands and face
recognition, gestures can create a richer user experience that strives to understand the human
"language," thereby fueling the next wave of electronic innovation.

Fig-3.9-Receiving phone calls

Fig-3.10-Waving the hand

28
When most people think of gesture recognition, they often imagine someone waving his or
her hands, arms or bodies in the effort to control a game or other application on a large-screen
display. Case studies of this trend include Microsoft’s Kinect peripheral for the Xbox 360,
along with a range of gesture solutions augmenting traditional remote controls for televisions
and keyboards, mice, touchscreens and trackpads for computers. At the recent Consumer
Electronics Show, for example, multiple TV manufacturers showcased camera-inclusive
models that implemented not only gesture control but also various face recognition-enabled
features. Similarly, Intel trumpeted a diversity of imaging-enabled capabilities for its
Ultrabook designs.

However, gesture recognition as a user interface scheme also applies to a wide range of
applications beyond consumer electronics. In the automotive market, for example, gesture is
seen as a convenience-driven add-on feature for controlling the rear hatch and sliding side
doors. Cameras already installed in rear of the vehicle for reversing, and in the side mirrors
for blind spot warning, can also be employed for these additional capabilities. As the driver
approaches the car, a proximity sensor detects the ignition key in the pocket or purse and
turns on the cameras. An appropriate subsequent wave of the driver’s hand or foot could
initiate opening of the rear hatch or side door.

Another potential automotive use case is inside the cabin, when an individual cannot (or at
least should not) reach for a particular button or knob when driving but still wants to answer
an incoming cellphone call or change menus on the console or infotainment unit. A simple
hand gesture may be a safer, quicker and otherwise more convenient means of accomplishing
such a task. Many automotive manufacturers are currently experimenting with (and in some
cases already publicly demonstrating) gesture as a means for user control in the car, among
other motivations as an incremental safety capability.

Additional gesture recognition opportunities exist in medical applications where, for health
and safety reasons, a nurse or doctor may not be able to touch a display or trackpad but still
needs to control a system. In other cases, the medical professional may not be within reach of
the display yet still needs to manipulate the content being shown on the display. Appropriate
gestures, such as hand swipes or using a finger as a virtual mouse, are a safer and faster way
to control the device .

29
 Fig 3.11-Useful in medical treatment
Gesture interfaces are also useful in rehabilitation situations. Gesturetek’s IREX, for
example, guides patients through interactive exercises that target specific body parts. And
less conventional health-related applications for gesture recognition also exist. For instance,
users with physical handicaps may not be able to use a conventional keyboard or mouse but
could instead leverage the recognition of facial gestures as a means of control. And active
university research is also underway on using gesture recognition as a means of translating
sign language to text and speech.

3.3 DISADVANTAGES

-Key limitation to this technology is that it only recognises motions, such as a hand flick or
circular movement, within a six-inch range.

-Effectiveness of the technology in adverse light conditions

-Variations in the background, and high power consumption.

-Image and background noise

-Resolution and quality

-Several challenges remain for gesture recognition technology for mobile devices, including
effectiveness of the technology in adverse light conditions, variations in the background, and
high power consumption.

However, it is believed these problems can be overcome with different tracking solutions and
new technology.

30
 CHAPTER-4

PRESENT SCENARIO

4.1 EVOLUTION OF USER-INTERFACES

We’ve come a long way in terms of both – the evolution of mobile phones as well as
methods to use and control these electronic devices, which have always been some kind of a
challenge.

It started out with those bulky phones, which incorporated long antennas, physical switches
and buttons; for example the Analog Motorola Dyna 8000X. Following this, the next round
of innovation pushed touch-sensitive interaction with track-pads like the IBM Simon for
example and currently its the touch-screen interface – the most dominating and familiar
interface, for cellular phone users around the world. In the meantime, there was work being
done to create voice-interaction with these electronics devices, which as a matter of fact has
also come a long way, but is still not robust enough and can’t cover all of the needs we have.

4.2 JUMPING FROM TOUCH-SCREENS TO GESTURE

RECOGNITION TECHNOLOGY

Over the past two years, we have seen smartphones explode in their use and
capabilities. From entertainment, to work or even to family duties, we now rely on these
devices in almost every aspect of our lives. There is, however, still room for our experience
with them to be enhanced. Touch was the last great user-interface advancement and hand
gesture recognition for mobile phones will be the next.Gesture recognition is perceived today
as the natural evolution of intuitive user interfaces. Since the creation of touch screens,
gestures have reigned in an entirely new aspect as to how we interact with our devices.
Gestures allow users to perform specific tasks in an extremely efficient and more dynamic
manner.

Some of the most used gestures are swipe to unlock, pinch to zoom and pull to refresh. While
those are relatively basic by most means, gestures have evolved greatly. There are a number
of different companies trying to push forward with touch-free gesture controls. Third party

31
developers have began to truly utilise the potential that multi-touch displays hold, all within
their apps. Gestures can offer an intuitive way to interact with a mobile phone. It is almost
similar in concept to Microsoft’s Xbox Kinect but on a much smaller scale.

4.3 ADDING A WHOLE NEW DIMENSION TO MULTIMEDIA

By adding touch-free functionality to smartphones, there’s a brand new dimension to

mobile games and applications. We can now take advantage of the Z-axis by using simple
hand shapes to enable reality-like experiences. A few examples of supported actions in games
include bowling, throwing darts, playing rock-paper-scissors and more.

Other scenarios made easier with hand gesture recognition are controlling the phone with
dirty or wet hands, managing calls while driving or working and interacting with a
smartphone when wearing gloves. Mobile operating systems like Android also find their way
to the big screen like TV and with that, this new experience will be offered to a broader
audience on a wider range of devices, controlling you TV with your hands with no need for a
remote control or an external 3D camera is the kind of freedom consumers are looking for.

“Interacting with consumer electronics is about to take another giant leap forward as hand
gesture recognition is increasingly incorporated into our smartphones, tablets, laptops, all-in-
ones and more,” said Assaf Gad, vice president of marketing for PointGrab. “It’s an exciting
time for both OEMs and end-users, as our devices are becoming even more intuitive and
entertaining.”Korean phone maker Pantech, best known in the U.S. for low-end Android and
messaging phones, has added gesture-based controls to its Vega LTE handsets to use Kinect-
Like Gesture Recognition in Android Phones. By incorporating this recognition software, it
allows users to control the handset with gestures. The technology, which comes from Israel’s
eyeSight Mobile Technologies, is a part of the Vega LTE

4.4 LATEST DEVICES

- Dell has created a beautiful Android Tablet with the $400 Venue 8 7000
Series.Released on Jan 12, 2015 at the 2015 International CES, the big trade show in
Las Vegas.
Special features are- (i) Camera can recognize the gestures
(ii) Dell Gallery also let you find photos using face recognition.
- LG G Flex 2 to hit market on 29 Jan,2015.

32
 Sold to the Korean market at around USD$738.
64 bit octa-core Qualcomm Snapdragon 810 processor
Two GB RAM, Android Version – 5.0 Lollipop
Protected by Gorilla Glass 3
Perfect selfie feature called Gesture Shot
- Hisense- new smartphones and tablet scheduled for launch in South Africa in March
this year. The smart-phone is named Infinity 6 and the tablet- Android Sero 8.
The Android version used is 4.4

33
 CONCLUSION & FUTURE SCOPE

Android offers a fresh take on the way mobile application interact with users, along with
the technical underpinnings to make it possible. It combines the ubiquity of cell phones, the
excitement of open source software , and the corporate backing of Google and other Open
Handset Alliance members. Touch screens are a great way to interact with applications on
mobile devices. Here by using our technology we can recognise Malayalam gestures which
are not yet developed. The Android framework makes it's easy to recognize simple action. As
touch screens are now one of the most common ways of interacting with computers, they can
do so effectively and efficiently in the recognition of human gestures. This work provides
new information about how Malayalam gestures can be incorporated in a gesture recognition
system even though it contains huge amount of alphabets and also symbols. We believe that
this work will bring us closer to the creation of robust and usable touch screen interfaces that
work equally well for all classes of people.

Gestures can be used to perform particular actions. Gesture application can be integrated with
other applications- like a flip to move the next window , a swirl to delete files .Here for
getting a song or a picture just a simple touch is needed .Future surgeons may use Robotic
nurse “gesture recognition”

Another important future application is MSIS-Multi semantic interpretation system .It is the
combination of all gestures(3D) and sounds. If this technology comes into the market it will
create revolutionary changes in all the fields.

34