Internet of Things (IoT) and Human Computer Interaction

Olja Rastic-Dulborough
School of Electronics and Computer Science
University of Southampton
Southampton, United Kingdom
[email protected]

ABSTRACT computing, in the paper co-authored with Brown; “The coming

This paper outlines the current state of the Internet of Things from age of calm technology” [11]
the human-computer interaction point of view.

Categories and Subject Descriptors

C.2.1 Network Architecture and Design, Network communications

General Terms
Design, Experimentation, Human Factors

Keywords
Internet of Things, Human-Computer Interaction

1. INTRODUCTION
Internet of things can be introduced as the concept where objects Figure 1 1996 view on trends in computing [11]
in our environment, through some new properties, become smart Some 20 years later, the reality is somewhat more complex, and
and begin autonomously communicating with each other and we’re still grasping at the edges of ‘calming’ the technology that
humans, through networks supported by interfaces. Opportunities surrounds us. Looking at the prognosis for the next decade and
for meaningful benefits to humans and the environment are many beyond, Figure 2 illustrates that we are in the early stages of
and varied so only a small subset of examples can be mentioned seamlessly integrating computing into our lives.
here. Specifically, the focus of this paper is to present the recent
efforts and developments in the areas of interactive systems, with
new affordances of the Internet of things and related technologies; 3. INTERACTION RESEARCH FOR IoT
emergence of and need for new and innovative ways in which Apart from architecture, physical device, security and protocol
human-computer interaction can support these. In section two concerns, amongst others, there are many human-centred issues
some background is given, section three looks at different being addressed. One looks at how best to present the data
approaches researchers are taking to interaction for the internet of collected from the real world and make it meaningful to humans.
things. Section four reviews the research, conclusions made in We look at several approaches to solving the interaction
section five. complexities in the Internet of Things.
3.1 Embedded interaction
2. BACKGROUND AND RELATED WORK Embedded interaction is a concept where ordinary objects are
15 years since Internet of Things as a term was first mentioned, enhanced with input and output facilities so that they perform
the field is entering a new phase of rapid expansion in all areas of their original function as well as providing a way for data to be
business, industry, domestic and personal devices. Building on a collected about the user behaviour. Combining data from several
networked device’s ability to be uniquely identifiable, which can objects and interaction with them over time can result in emergent
communicate and interact with other devices, objects and humans knowledge available to the user or the context aware environment
[8]; the combination of research and development efforts both in to make suggestions for lifestyle improvements.
academia and industry has opened up new avenues of exploration
A study carried out by Kranz, Holleis and Schmidt [6]
for how the ubiquitous connectivity can enhance our every-day
concentrated on designing prototypes for novel ways humans can
lives. Because of the organic growth and development by different
use everyday objects incorporating embedded interaction and
stakeholders, multiple standards are in existence. Although there
technologies which take advantage of the Internet of things
are efforts to standardise protocols, guidelines and processes; it is
affordances. They coin the term ‘netgets’ to describe objects or
recognised that doing this too early would restrict the
artefacts as specialized networked gadgets with sensors and
development along many lines of investigation. Instead, it is
actuators which can be embedded into anything. Netgets let users
interesting to take stock of the concerns and the state of things as
seamlessly manipulate digital information and data in the context
Xie, Kumar, Chen at all [12] have done. Going back to the early
of real world usage. Examples range from context aware kitchen
days when ubiquitous computing was first being defined, the table
utensils, interactive fabrics to using small embedded objects.
in Fig 1 neatly summarises the realization of the ‘father of
Along the way, guidelines, frameworks and a toolkit are produced
ubiquitous computing’ Mark Weiser about the future of
and recommended for future development of similar devices.
Figure 2 Recent predictions for the IoT domains and stages by Xie, Kumar, Chen et al [12]

3.1.1 Smart interactive kitchen

One example from the study [6] is that of a chopping knife which
has sensors not only to detect the type of food being cut but can
also be used on the movement aware chopping board as a mouse /
pad combination to control a computer.

Figure 4 Smart kitchen environment

When data from other elements of this smart kitchen environment
such as a microphone, camera and body-worn sensors are
collected and contextualised, it enables the digital system to
suggest ingredients to be added based on the ones being detected
Figure 3 Smart kitchen elements as well as reminders and suggestions to the users about their
nutrition.
There is already an implementation of a similar idea on the market
– a digital fork called HapiFork1. It records the length of the meal,
counts fork servings per minute and intervals between fork 3.1.3 Small embedded objects
servings. The premise is that by being aware of the speed of the Another of the case studies conducted by Kranz, Holleis and
meal, the user can control it, slow it down and in the process lose Schmidt [6] is one using the small embedded objects in the
weight. The data from the device is uploaded to the user’s online physical world. Their prototype to establish room availability by
dashboard via USB or Bluetooth and the information presented to using a small magnet placed appropriately on the board outside of
the user is also available on the mobile devices. Meal statistics are the door of a room, which is detected by a hall sensor provides
displayed and recommendations or cautions provided based on information about room occupancy in a very simple manner.
real time data.
While both the research and the marketed product have interesting 3.2 Physical mobile interaction (PMI) and
implications for the future of interaction, there is still the question
multi tag interaction MTI)
of how ready the users are to engage with and invest in these
Broll, Rukzio, Paolucci et al [1] collaborated on creating a system
products. From the viewpoint of learning about our own
for mobile ticketing, for cinema and transportation, which users
behaviour, it might be wise to introduce individual solutions to
interact with on their mobile devices and smart posters (PMI) with
our home environments, one by one, as they become available, as
different types of tags provided for each option and step through
to be exposed to the fully integrated smart home in one move
processes (MTI). The three types of tags used were numeric
might be intimidating and have the feel of the “big brother”
identifiers, Near Field Communication (NFC) tags and visual
watching.
markers. Users get the information from the tags and use it to
reach desired services. The study focused on evaluating user’s
3.1.2 Wearable touch input clothing reactions and experience using the step through process of
The same team Kranz, Holleis and Schmidt [6] created the interacting with multiple tags to reach a desired goal. The main
Embedded Interaction Toolkit2 which enables them to quickly use barrier to users was the non-familiarity with the system of tags
and connect smart objects and has flexibility for changing the and interacting with smart posters in this new way.
protocols for each individual type of object, as well as how the
messages travel between the objects and platform. With the help Most posters and leaflets these days have QR (Quick Response)
of EiToolkit they were able to prototype and test touch input on codes and the action of scanning them with the appropriate
clothing. application on a smartphone usually takes users to a website
which they can interact with on the smartphone’s browser.
The idea of interactive clothing through sensors and displays is
slowly increasing in use through the maker movement and moving Embedding more complex interaction into posters seems
into mainstream. unexplored and underutilised for such a cheap and widely
applicable method of distribution.
An example of note in this category is the Poken6 idea, it started
off as a USB stick with inbuilt NFC (Near Field Communication)
technology on which the users’ social media accounts are stored,
ready for quick exchanging with other Poken users. Through the
online interface, the account could be updated and interactions
with other users are tracked in a timeline. The service is now
updated for mobile application use and various crowd
management plans are on offer to organisers, including smart
posters, Poken dongles etc.

Figure 5 Turn signal bike jackets with LilyPad and lights

One example is the LilyPad3 electronic sewable kit offered by
Arduino4 Many small wearable components can be easily sourced
and sewing and programming instructions and projects abound on
the internet. An interesting and useful example is the Turn Signal
Bike Jacket5 which allows bicycle riders to have left or right
indicators on their backs, controlled by a small switch sewn into
the sleeves which can easily be operated by the rider without
disrupting their attention from the road.
1
HapiFork website - http://www.hapi.com/products-hapifork.asp
2
EiToolkit site - https://wiki.medien.ifi.lmu.de/HCILab/EiToolkit
3
Lilypad website - http://lilypadarduino.org/
4 Figure 6 Dashboard for Poken account
Arduino website - http://arduino.cc/
5 An example use would be at a conference, where attendees would
Turn Signal Jacket -http://mocoloco.com/fresh2/2010/02/11/turn-
signal-bike-jacket-by-leah-buechley.php
6
Poken website - http://www.poken.com/
be sent details of the app in advance and on the day, they could the environment can be controlled remotely via a mobile
easily be guided through the conference proceedings, their application and the web. They introduce a metaverse server and
interactions with smart posters and other attendees recorded; with client that communicate with the home server; the devices self-
analysis tools available to the organisers and contact details and report their status which then updates the animation in the virtual
presentations available to attendees, after the event. world with which the user interacts.
This is a visualisation of the state of things in an environment and
3.3 Augmented reality approach although the user can control things through an avatar moving
Gimenez and Pous [4] offer a view on issues in interaction by through the virtual world, I imagine this would become tiresome
humans in complex smart environments, and look for most after the first few times and when the novelty of having to play a
effective ways of connecting those environments and people. game to switch the lights on or close the curtains. An option to be
They put forward an argument that augmented reality offers best able to switch back to a non-gamified interface would be
solutions in such cases, especially where geographically relevant desirable.
information is available and can be useful to the user and
conclude that the move away from Graphical User Interfaces to A study where a similar idea of a virtual environment is used,
Tangible User Interfaces will improve Human-Computer albeit as video taken of a real environment rather than a
Interaction. Augmented Reality is key in providing a technology constructed graphical representation, is the MIT Media Lab study
supported layer to the existing physical environment. [9] [3] of how contextual referencing of language and speech
development in relation to location and social interaction within
Space Top [7] is an interesting example of an augmented reality one family was analysed through data collected over time. The
device created to enable interaction with existing objects and concept was tested further in different environments, and
concepts but in a new way which can easily lend itself to future extended in the TV Genome tool for use on sentiment analysis of
use for the Internet of things projects. tweets [10] with a common topic, for example, during a TV
programme where spectators would share their thoughts about
something a lot of other people were watching.

Figure 8 TV Genome platform for analysing event tweets

The difference here is that a new way of connecting people with
Figure 7 Space Top spatial operating environment things or even virtual things is emerging and creating the tools
The user is able to interact with virtual objects such as files and allows them to interact in a way they weren’t able to and the
windows, which appear through the 3D spatial operating researchers to offer insights (or advertising) based on their
environment Figure 7 above. analyses.

A current offering available to smartphone users, the app Layar

connects images, links and videos in the digital world to the real
4. INTERACTION AND IoT – A REVIEW
Some of the studies and working examples presented in section
objects in the physical world by displaying hidden content when
three touch in specific ways on how different stakeholders in the
looked at through the screen of the mobile phone, with the help of
Internet of things domain are exploring possibilities and concepts
the Layar app. Their main area of business is to develop products
to take advantage of the new emerging affordances made possible
like AR widgets for websites or solutions for printed materials,
by ubiquitous connectivity, improved processing and storage
enriching them with digital media. The app just enables
capacities as well as new display technologies, sensor device
consumers to access the resulting products.
availability and decreasing hardware costs.
This is a popular area for research with a lot of the solutions being Primary goals include creating a link between the physical and
explored to enhance what we see and know about the real world. digital worlds, embedding interaction behaviours into existing
objects, providing meaning from large amounts of data gathered
3.4 Virtual Reality approach and eventually enabling us to learn about ourselves, others and
Immersing the user more into digital world, Han, Yun, Jang and our surroundings.
Park [5] present a visually accurate 3D Virtual Reality Purely from interaction design perspective, away from technical
representation of the real world space in which the real devices and academic research, the Internet of things is causing
and excitement. In their short documentary Connecting [2] the design
7
Layar website - https://www.layar.com/magazine/ agency Bassett & Partners showcase thoughts from the industry
leaders on
 Figure 9 Hype Cycle of emerging technologies via Xie, Kumar, Chen et al [12]

the future of Interaction Design and User Experience for the

6. REFERENCES
Internet of things. The affordances of the internet of things give us [1] Broll, G., Rukzio, E., Paolucci, M., Wagner, M.,
an opportunity to become creators instead of consumers of data Schmidt, A. and Hussmann, H. 2009. Perci: Pervasive
and information. The by-product of intensive connectivity is a Service Interaction with the Internet of Things. IEEE
super organism where humans are not at the top of the food chain. Internet Computing. 13, 6 (Nov. 2009), 74–81.
We can move through our physical world and see traces and
footprints of our friends who walked along the same street at a [2] Connecting: 2012. http://vimeo.com/52861634.
different time, from the data available about the location, only the Accessed: 2013-12-03.
relevant information is made available to us through intelligent
[3] Fleischman, Michael Ben/Adviser-Roy, D. 2008.
analysis and selective filtering.
Grounding language in events. Massachusetts Institute
Looking at the Hype cycle of emerging technologies – Figure 9 – of Technology.
we are well on the way there.
[4] Gimenez, R. and Pous, M. 2010. Augmented reality as an
enabling factor for the Internet of Things. W3C
Workshop: Augmented Reality on the Web Mobile 2.0
5. CONCLUSION Europe (Barcelona, 2010).
Internet of things is an exciting area for research, particularly
when it comes to human computer interaction because the number
[5] Han, J., Yun, J., Jang, J. and Park, K.-R. 2010. User-
of use cases as well as the devices and systems which the user has friendly home automation based on 3D virtual world.
to interact with is constantly growing. Affordances created by the 2010 Digest of Technical Papers International
Internet of things technologies offer a glimpse into the future Conference on Consumer Electronics (ICCE) (Jan.
where the existing, familiar interactions of the past are still 2010), 499–500.
possible but are superseded by the smart capabilities of the same
objects which have new properties and affordances. How [6] Kranz, M., Holleis, P. and Schmidt, A. 2010. Embedded
comfortable we might be with a smart knife and a sensing Interaction: Interacting with the Internet of Things. IEEE
chopping board remains to be seen but this might not be Internet Computing. 14, 2 (Mar. 2010), 46–53.
important. More relevant are the other findings, recommendations
and discoveries made by the researchers which can then be [7] Lee, J., Olwal, A., Ishii, H. and Boulanger, C. 2013.
applied in any other fields or areas, within or outside of the realm SpaceTop. Proceedings of the SIGCHI Conference on
of the Internet of Things. Human Factors in Computing Systems - CHI ’13 (New
York, New York, USA, Apr. 2013), 189.
[8] Miorandi, D., Sicari, S., De Pellegrini, F. and Chlamtac, [10] Talbot, D. 2011. A Social-Media Decoder. MIT
I. 2012. Internet of things: Vision, applications and Technology Review.
research challenges. Ad Hoc Networks. 10, 7 (Sep. 2012),
1497–1516. [11] Weiser, M. and Brown, J.S. 1997. The coming age of
calm technolgy. (Jan. 1997), 75–85.
[9] Roy, D., Patel, R., DeCamp, P., Kubat, R., Fleischman,
M., Roy, B., Mavridis, N., Tellex, S., Salata, A., [12] Xie, B., Kumar, A., Chen, C.-M., Pandey, S., Nepal, S.,
Guinness, J., Levit, M. and Gorniak, P. 2006. Symbol Gubbi, J., Buyya, R., Marusic, S. and Palaniswami, M.
Grounding and Beyond. Springer Berlin Heidelberg. 2013. Internet of Things (IoT): A vision, architectural
elements, and future directions. Future Generation
Computer Systems. 29, 7 (2013), 1645–1660.