THE SIXTH SENSE TECHNOLOGY

Mandar Ghate
Department Of Computers, Padmabhushan Vasantdada Patil Pratisthans College Of Engineering, Mumbai.
[email protected]
Contact no. 8082495498/9029019950.

Abstract— Sixth Sense is a wearable gestural interface that augments objects or even people and project out information regarding what
the physical world around us with digital information and lets us use you are looking at.
natural hand gestures to interact with that information. Steve Mann is Sixth Sense Technology is a perception of augmented reality
considered as the father of Sixth Sense Technology who made concept. Sixth Sense is in fact, about comprehending information
wearable computer in 1990. He implemented the Sixth Sense more than our available senses. And today there is not just this
Technology as the neck worn projector with a camera system (which physical world from where we get information but also the digital
Mann originally referred to as “Synthetic Synesthesia of the Sixth world which has become a part of our life. This digital world is now
Sense”). He was a media lab student at that time. Then his work was as important to us as this physical world. And with the internet the
carried forward by Pranav Mistry (Ph.D student in the Fluid digital world can be expanded many times the physical world. God
Interfaces Group at the MIT Media Lab). By using a camera and a hasn’t given us sense to interact with the digital world so we have
tiny projector, Sixth Sense sees what you see and visually augments created them like smartphones, tablets, computers, laptops, net books
any surfaces or objects we are interacting with. It projects and other gadgets. These gadgets enable us to communicate with the
information onto surfaces, walls and physical objects around us, and digital world around us.
lets us interact with the projected information through natural hand But we are humans and our physical body isn’t meant for digital
gestures, arm movements, or our interaction with the object itself. world so we can’t interact directly to the digital world. For instance
This wearable gestural interface attempts to free information from its we press keys to dial a number; we type text to search it and so on.
confines by seamlessly integrating it with reality, and thus making This means for an individual to communicate with the digital world
the entire world your computer. To a layman, it would be something he/she must learn it. We don’t communicate directly and efficiently
supernatural. Some might just consider it as a superstition or to the digital world as we do with the real world. The Sixth Sense
something psychological. But the invention of Sixth Sense Technology is all about interacting to the digital world in most
Technology has completely shocked the world. Although it is not efficiency and direct world.
widely known as of now but the time is not far when this technology
will change our perception of the world. COMPONENTS

INTRODUCTION The devices which are used in Sixth Sense Technology are:

We've evolved over millions of years to sense the world around us. 1. Camera.
When we encounter something, someone or some place, we use our 2. Coloured Marker.
five natural senses to perceive information about it; that information 3. Mobile Component.
helps us make decisions and chose the right actions to take. But 4. Projector.
arguably the most useful information that can help us make the right 5. Mirror.
decision is not naturally perceivable with our five senses, namely the
data, information and knowledge that mankind has accumulated
about everything and which is increasingly all available online.
Although the miniaturization of computing devices allows us to carry
computers in our pockets, keeping us continually connected to the
digital world, there is no link between our digital devices and our
interactions with the physical world. Information is confined
traditionally on paper or digitally on a screen. Sixth Sense bridges
this gap, bringing intangible, digital information out into the tangible
world, and allowing us to interact with this information via natural
hand gestures. Sixth Sense Technology is the newest jargon that has
proclaimed its presence in the technical arena. This technology has
emerged, which has its relation to the power of these six senses. Our
ordinary computers will soon be able to sense the different feelings
accumulated in the surroundings and it is all a gift of the “Sixth
Sense Technology” newly introduced. Sixth Sense will allow us to
interact with our world like never before. We can get information on 1.Camera:
anything we want from anywhere within a few moments! We will It captures the image of the object in view and tracks the user’s hand
not only be able to interact with things on a whole new level but also gesture. The camera recognizes individuals, images, pictures,
with the people! One great part of the device is its ability to scan gestures that user makes with his hand. The camera then sends this
data to a smartphone for processing. Basically the camera forms a 4. The information that is interpreted through the smartphone can be
digital eye which connects to the world of digital information. projected into any surface.
2. Coloured Marker: 5. The mirror reflects the image on to a desired surface.
There are colour markers placed at the tip of users fingers. Marking
the user’s fingers with red, yellow, green and blue coloured tape
helps the webcam to recognize the hand gestures. The movements
and arrangement of these markers are interpreted into gestures that
act as a interaction instruction for the projected application
interfaces.

3. Mobile Component:
The Sixth Sense device consists of a web enabled smartphone which
process the data send by the camera. The smartphone searches the
web and interprets the hand gestures with the help of the coloured
markers placed at the finger tips. Basic processing works on
computer vision algorithms where approx. 50,000 lines of code are
used written in Symbian C++.

4. Projector:
The information that is interpreted through the smartphone can be
projected into any surface. The projector projects the visual
information enabling surfaces and physical objects to be used as
interfaces. The projector itself consists of a battery which have 3
hours of battery life. A Tiny LED projector displays the data sent
from the smartphone on any surface in view-object, wall or person.
The downward facing projector projects the image on to a mirror.

5. Mirror: APPLICATIONS
The usage of a mirror is important as the projector dangles pointing
downward from the neck. The mirror reflects the image on to a desire The Sixth Sense Technology finds a lot of applications in the modern
surface. Thus finally the digital image is freed from its confines and world. The Sixth Sense devices bridge the gap by bringing yhe
placed in the physical world. digital world into the real world and in the process allowing the users
to interact with the information without the help of any machine
KINDS OF GESTURES RECOGNIZED interfaces. Prototypes of the Sixth Sense device have demonstrated
viability, usefulness and flexibility of this new technology.
The software recognizes three kinds of gestures: Acoording to the words of its developers the extend of use of this
new device is only limited bythe imagination of human beings.Some
1.Multi-Touch Gestures: practical apllications of the Sixth Sense Technology is given below:
Like the ones we see in the iphone-where we touch the screen and
make the map move by pinching and dragging. 1.Viewing Map:
With the help of a map application the user can call upon any map of
2. Freehand Gestures: his/her choice and navigate through them by projecting the map on to
Like when you take a picture or Namaste gesture to start the any surface. By using the thumb and index fingers movements the
projection on the wall. user can zoom in, zoom out or pan the selected map.

3. Iconic Gestures:
Drawing an icon in the air. Like, whenever we draw a star, shows us
the weather details. When we draw a magnifying glass, shows us the
map.

WORKING

The Sixth Sense Technology works as follows:

1.It captures the image of the object in view and track the user’s hand
gestures.
2. There are colour markers placed at the tip of users finger. Marking
the user’s fingers with red, yellow, green and blue coloured tape
helps the webcam to recognize the hand gestures. The movements
and arrrangement of these markers are interpreted into gestures that
act as a interaction instruction for the projected application
interfaces. 2. Taking Pictures:
3. The smartphone searches the web and interprets the hand gestures Another application of Sixth Sense devices is the implementation of
with the help of the coloured markers placed at the finger tips. a gestural camera. This camera takes the photo of the location user is
looking at by detecting the framing gesture. After taking the desired
number of photos we can project them onto any surfaces and then use
gestures to sort through those photos and organize and resize them.

6. Getting Information:
Sixth Sense devices can be used for getting various information
relating to our everyday life by getting in contact with objects.

(a)Product information:
3. Drawing Application: Sixth Sense Technology uses marker technology or image
The drawing application allows the user you to draw on any surface recognition techniques to recognize the objetcs we pick in our hand
by tracking the fingertip movements of the user’s index finger. The and then provide information realting to the product.
pictures that are drawn by the user can be stored and replaced on any
other surface. The user can also shuffle through various pictures and (b) Book Information:
drawing by using the hand gesture movements. By holding and shuffling through the book pages, the Sixth Sense
provides Amazon ratings on that book, other reviews and other
4. Making Calls: relevant things related to the book.
We can make calls with the help of Sixth Sense device. The Sixth
Sense device is used to protect the keyboard into your palm and (c) Flight Updates:
using that virtual keypad we can make calls to anyone. With the help of the Sixth Sense Technology it is no longer required
to log into any sites for checking the status of the flights. The system
will recognize your boarding pass and let you know whether the
flight is on time or not.

5. Interacting with physical objects:

The Sixth Sense system also helps to interact with physical objects
we use in a better way. It augments physical objects by projecting
more information about these objects projected on them. For
example, a gesture of drawing a circle on the user’s wrist projects an
7. Sixth Sense also lets the user draw icons or symbols in the air
analog watch on the user’s hand. Similarly a newspaper can show
using the movement of the index finger and recognizes those
live video news or dyanamic information can be provided on a
symbols as interaction instructions.
regular piece of paper.
For example, drawing a magnifying glass symbol takes the user to
the map application or drawing an ‘@’ symbol lets the user check his
mail.

ADVANTAGES

1.Portable:
One of the main advantages of the Sixth Sense devices is its small
size and portability. It can be easily carried around without any
difficulty. The prototype of the Sixth Sense is designed in such a way
that it gives more importance to the portability factor. All the devices
are light in weight and the smartphone can easily fit into the user’s
pocket.

2. Support Multi touch and Multi user interaction:

Multi touch and Multi user interaction is another added feature of RELATED TECHNOLOGIES
the Sixth Sense devices. Multi sensing techique allows the user to
interact with system with more than one finger at a time. Sixth Sense Gesture recognition:
devices also in-corporates Multi user functionality. This is typically It is a technology whichis aimed at interpreting human gestures with
useful for large interaction scenarios such as interactive table tops the help of mathematical algorithms. Gesture recognition technique
and walls. basically focuses on the emotion recognition from the face and hand
getsure recognition. Gesture recogniton technique enables humans to
3. Cost Effective: interact with computers in a more direct way without using any
The cost incurred for the construction of the Sixth Sense prototype is external interfacing devices. It can provide a much better alternative
quite low. It was made from parts collected together from common to text user interfaces and graphical user interface which requires the
devices. And a typical Sixth Sense device costs upto $300. The Sixth need of a keyboard or mouse to interact with the computer.Interfaces
Sense devices have not been made in large scale for commercial which solely depends on the gestures requires precise hand posing
purpose. Once that happens it’s almost certain that the device will tracking. In the Sixth Sense devices coloured vands are used for this
cost much lower than the current price. purpose. Once hand pose has been captured the gesture’s can be
recognised using different technique’s. Neural network approaches or
statistical templates are the common techniques used for the
4. Data access directly from the machines in real time: recognition purposes. This technique have an accuracy of more than
With the help of a Sixth Sense device the user can easily access data 95%. Time dependent neural network will also be used for real time
from any machine at real time speed. The user doesn’t require any recognition of the gestures.
machine-human interface to access the data. The data access through
recognition of hand gestures is much easier and user friendlier
compared to the text user interface or grapichal user interface which Augmented reality:
requires keyboard or mouse. The Augmented reality is a visualiztaion technique that allows the
user to experience the virtual experience added over real world in
real time. With the help of advanced AR technology the information
5. Mind map the idea anywhere: about the surrounding real world of the user becomes interactive and
With the advent of the Sixth Sense device, requirement of a platform digitally usable. Artificial infromation about the environment and the
or a screen to analyze and interpret the data has become obsolete. We objetcs in it can be stored and retrieved as an information layer on
can project the information onto any surface and can work and top of the real world view. When we compare the spectrum between
manage the data as per our convenience. virtual reality, which creates immersive, computer-generated
environments, and the real world, augmented realtiy is closer to the
6. Open Source Software: real world. Augmented reality adds graphics, sounds, haptic feedback
The software that is used to interpret and analysis the data collected and smell to the natural world as it exists. Both video games and cell
by the device is going to be made open source as said by its inventor. phones are driving the development of augmented realtiy. The
This will enable other developers to contribute to the development of augmented systems will also supreimpose graphics for every
the system. perspective available and try adjust to every movement of the user’s
head and eyes. The three basic componenets of na augmented reality
CONCERNS system are the head mounted display, tracking system and mobile
computer for the hardware. The main goal of this technology is to
merge these three components into a highly poratble unit. The head
There is a health issue regarding Sixth Sense’s projection technology. mounted display used in augmented realtiy systems will enable the
When the device is projecting on a hard surface, it is not private user to view superimposed graphics and text created by the system.
enough just for the user. People around him can see the projection Another component of augmented reality system is its tracking and
that is very detailed. Projection is better in the night time and dark orientation system. This system pinpoints the user’s location in
areas rather than mornings and bright areas. This is an issue because reference to his surrooundings and additionally tracks the user’s eye
the vision of the user can be damaged. Sixth Sense should be able to and head movements.
shift its projection techniques during different times of the day. That
way it wont be an issue for the vision of the user. Since the device is
still being modified and tested, Mistry can try to overcome concerns Computer vision:
with projection. Computer vision is the technology in which machines are able to
Concerns about the pricing of this device is also rising mong the interpret/extract necessary information from an image. Computer
people. Mistry declared way back in 2009 that this device would cost vision technology includes various fields like image processing,
$350 but ever since then there is no news about this device. Mistry is image analysis and machine vision. It includes certain aspect of
working on many other technologies and invetions, but the world artificial intelligence techniques like pattern recognition. The
doesn’t know whether the work on this device has stopped. People machines which implement computer vision techniques requires
and manufacturers who are ready to have this device do no know image sensors which detect electromagnetic radiation which are
when this will be out. usually in the form ultraviolet rays or light rays. The computer vision
find itself applicable in various fields of interest. One such field is
bio medical image processing. It’s also used in autonomous vehicles
like SUV’s. The computer vision technique basically includes four
processes.
1.Recogniton: One of the main task of computer vision technique is augmenting the physical world with digital information, bringing
to determine whether the particular object contain the useful data or intangible information into tangible world.
not. Sixth Sense recognizes the objects aroound us, displaying
2. Motion Analysis: Motion analysis includes several tasks related to infromation automatically and letting us access it in any way we
estimation of motion where an image sequence is processed need. The Sixth Sense prototype implements several applications that
continuously to detect the velocity at each point of the image or int demonstrate the uefulness, viability and flexibilty of the system. The
the 3D scene. potentil of becoming the ultimate “transparent” user interface for
3. Scene Reconstruction: Computer vision technique employs several accessing information about everything around us is present in this
methods to recreate a 3D image from the available images of a scene. technology. Using this technology “Finally one gets a taste of the
4. Image Restoration: The main aim of this step is to remove noise world of Harry Potter.” Sixh Sense Technology is the science of
from a given image. The simplest method involves using simple tomorrow with the aim of connecting the physical world seamlessly,
filters like low pass or median filters. In order to get better quality eliiminating hardware devices.
images more complex methods like Inpainting are used.
ACKNOWLEDGEMENT

In this paper, I would like to acknowledge Mr. Pranav Mistry for the
Radio Frequency Identification: great and deep research work he has done in this field.
Radio frequency identification systems transmit the identity of an
object wirelessly, using radio magnetic waves. The main purpose of a REFERENCES
radio frequency identification system is to enable the transfer of a
data via a poratble device. The portable device is commonly kknown 1. http://www.pranavmistry.com/projects/sixthsense
as a tag. The data send by the tag is received and processed by a
reader according to the needs of the application. The data send send
by the tag cotains varioous information’s identification or location of 2.http://news.softpedia.com/news/Next-Gen-039-Sixth-Sense-
the information, or specifies aboout the product that has been tagged, 039-Device-Created-at-MIT-103879.shtml
for example price, colour, date of purchase,etc. This technology
gained imporatance because of its ability to track moving object. A 3. http://gizmodo.com/5167790/sixth-sense-technology-may-
typical radio frequency tag consists of a microchip attached to a radio
change-how-we-look-at-the-world-forever
antenna which is mounted on a substrate. To retrieve data from the
tag a reader is needed. A typical radio frequency reader consists of
two antennas that emit radio waves and at the same time are capable 4.http://www.ted.com/talks/pattie_maes_demos_the_sixth_sen
of accepting the signals from the tag. The reader then passes the se.html
information that it has received to a computer device in digital
format. The computer then interprets this digital data and processes 5.http://theviewspaper.net/sixth-sense-technology-will-
it. Radio frequency identification tehniques are widely used in the
fields like asset tracking, supply chain management, manufacturing,
revolutionize-the-world/
payment systems,etc. One of the major advantages of devices using
radio frequency technology over other similar devices is that RFID 6.http://www.technologyreview.com/TR35/Profile.aspx?
devices need not be positioned precisely relative to the scanner. But TRID=816
till then there are certain areas of concern fot this technology. Some
problem realted to this technology is tag collosion and reader 7. http://news.bbc.co.uk/2/hi/technology/7997961.stm
collision. Usually the reader collision occurs when the signals from
the signals from two or more readesr overlap, while tag collision
occurs when many tags are present in a small area.
8.http://boingboing.net/2009/11/12/sixth-sense-technolo.html

9. Nikos Paragios and Yunmei Chen and Olivier Faugeras

FUTURE ENHANCEMENTS
(2005). Handbook of Mathematical Models in Computer
1.To get rid of color markers. Vision. Springer. ISBN 0-387-26371-3.
2. To incorporate camera and projector inside mobile computing
device. 10.Wilhelm Burger and Mark J. Burge (2007). Digital Image
3. To have 3D gesture tracking. Processing: An Algorithmic Approach Using Java. Springer.
4. To make Sixth Sense work as fifith Sense for disabled person.
ISBN 1846283795 and ISBN 3540309403.
http://www.imagingbook.com/.
CONCLUSION
11.Pedram Azad, Tilo Gockel, Rüdiger Dillmann (2008).
Information is often confined to paper or computer screens. Sixth
Sense frees data from these confines and seamlessly integrate Computer Vision - Principles and Practice. Elektor
inforation and reality. With the miniaturization of computing devices, International Media BV. ISBN 0905705718.
we are always connected to the digital world, but there is no link http://ivt.sourceforge.net/book.html.
between oour interactions with these digital devices and our
interactions with the physical world. Sixth Sense bridges this gap by
12. Reinhard Klette, Karsten Schluens and Andreas Koschan
(1998). Computer Vision - Three-Dimensional Data from
Images. Springer, Singapore. ISBN 981-3083-71-9.

13. Tony Lindeberg (1994). Scale-Space Theory in Computer

Vision. Springer. ISBN 0-7923-9418-6.
http://www.nada.kth.se/~tony/book.html.