Doug A.

Bowman
[email protected]
An Introduction to 3-D User
Department of Computer Science Interface Design
Virginia Polytechnic Institute and
State University

Ernst Kruijff
ernst.kruijff @archit.uni-weimar.de
Abstract
InfAR-computer science in
architecture
Bauhaus Universitaet Weimar Three-dimensional user interface design is a critical component of any virtual envi-
ronment (VE) application. In this paper, we present a broad overview of 3-D inter-
Joseph J. LaViola, Jr. action and user interfaces. We discuss the effect of common VE hardware devices
[email protected] on user interaction, as well as interaction techniques for generic 3-D tasks and the
Computer Science Department use of traditional 2-D interaction styles in 3-D environments. We divide most user-
Brown University interaction tasks into three categories: navigation, selection/manipulation, and system
control. Throughout the paper, our focus is on presenting not only the available
Ivan Poupyrev techniques but also practical guidelines for 3-D interaction design and widely held
[email protected] myths. Finally, we briey discuss two approaches to 3-D interaction design and
Sony Computer Science Lab some example applications with complex 3-D interaction requirements. We also
Tokyo, Japan present an annotated online bibliography as a reference companion to this article.

1 Introduction

User interfaces (UIs) for computer applications are becoming more di-
verse. Mice, keyboards, windows, menus, and icons—the standard parts of tra-
ditional WIMP interfaces—are still prevalent, but nontraditional devices and
interface components are proliferating rapidly. These include spatial input de-
vices such as trackers, 3-D pointing devices, and whole-hand devices allowing
gestural input. Three-dimensional, multisensory output technologies—such as
stereoscopic projection displays, head-mounted displays (HMDs), spatial audio
systems, and haptic devices—are also becoming more common.
With this new technology, new problems have also been revealed. People
often Žnd it inherently difŽcult to understand 3-D spaces and to perform ac-
tions in free space (Herndon, van Dam, & Gleicher, 1994). Although we live
and act in a 3-D world, the physical world contains many more cues for under-
standing and constraints and affordances for action that cannot currently be
represented accurately in a computer simulation. Therefore, great care must go
into the design of user interfaces and interaction techniques for 3-D applica-
tions. It is clear that simply adapting traditional WIMP interaction styles to
three dimensions does not provide a complete solution to this problem.
Rather, novel 3-D user interfaces, based on real-world interaction or some
other metaphor, must be developed.
This paper is a broad overview of the current state of the art in 3-D user in-
terfaces and interaction. It summarizes some of the major components of tuto-
Presence, Vol. 10, No. 1, February 2001, 96 –108 rials and courses given by the authors at various conferences, including the
© 2001 by the Massachusetts Institute of Technology 1999 Symposium on Virtual Reality Software and Technology. Our goals are

96 PRESENCE: VOLUME 10, NUMBER 1

 Bowman et al. 97

to describe some of the major interaction techniques devices. Fully immersive displays—such as head-
and interface components that have been developed for mounted displays (HMDs), arm-mounted displays, and
3-D systems, especially immersive virtual environments virtual retinal displays— occlude the real world. (Aug-
(VEs), and to provide the reader with some practical mented realty is an exception to this rule.) Because of
guidelines for the design of 3-D user interfaces in real- this, physical objects require a graphical representation
world applications. in the virtual world. In addition, more-complicated in-
It is impractical to list even a small fraction of the ref- put devices may be difŽcult to use because they cannot
erence materials from which this overview is taken, but be seen.
we refer readers to the “20t h Century 3DUI Bibliogra- Semi-immersive displays—such as stereo monitors,
phy” (Poupyrev & Kruijff, 2000), a continuously up- workbenches, and surround-screen virtual reality sys-
dated and annotated list of references that is available tems—allow the user to see both the physical and vir-
online. This bibliography was created both by the au- tual world. A number of interface issues arise with
thors and by other members of the 3DUI mailing list, semi-immersive displays. For instance, because users
an online community of researchers in this Želd. can see their own hands in front of the display, they
can inadvertently block out virtual objects that should
appear to be closer than their hands. This problem
2 Input and Output Devices
occurs frequently during manipulation tasks and can
be alleviated if the virtual object is offset from the
Input and output (I/O) devices are an important
hand so that it is always visible. Another problem re-
component in building 3-D user interfaces for VE appli-
lates to the use of stereo shutter glasses. Because
cations. Interaction designers must have a thorough
these require emitters, any line-of-sight occlusion will
understanding of the ergonomics, advantages, and limi-
disrupt the stereo effect. Interface designers should
tations of the devices used so they can Žnd natural and
try to avoid interaction techniques that cause users to
intuitive mappings between interaction techniques and
move their hands or physical objects in the line of
hardware.
sight of the glasses and emitters.
Literally hundreds of different I/O devices are used
In addition to visual displays, auditory, tactile, and
in VE applications, and a thorough analysis of them is
haptic displays are becoming more prominent in 3-D
beyond the scope of this paper. (See Youngblut, John-
son, Nash, Wienclaw, and Will (1996) for a somewhat user interfaces for VEs. With auditory displays (Be-
dated but thorough discussion on many I/O devices.) gault, 1994), the main interface goals are localiza-
However, many of these devices can be classiŽed into tion, the generation of 3-D sound, and soniŽcation
groups with common characteristics and interaction de- (transforming certain types of information into
sign criteria. In the following subsections, we examine sound). Auditory output is especially useful in collab-
these groups and discuss interface design issues within orative interfaces in which participants can get a sense
them. for where others are in the virtual world. It can also
be used to substitute for missing tactile or haptic
feedback. For example, a sound could substitute for
2.1 Output Devices
the feel of pressing a button. Haptic and tactile dis-
We commonly use the term display to describe plays (Burdea, 1996) are an important component of
output. Although the most common displays are visual, 3-D user interfaces for VEs, and this is becoming an
there are other important displays for the auditory, hap- active area of research. Allowing users to touch and
tic, tactile, and olfactory channels. In the context of 3-D feel in the virtual world can be extremely powerful,
user interfaces for VEs, visual displays can be roughly especially for object manipulation and collision
categorized into fully immersive and semi-immersive detection.
98 PRESENCE: VOLUME 10, NUMBER 1

2.2 Input Devices computer know when to listen to the user’s voice is one
such issue. A user may issue a speech command unin-
A distinction must be made between input devices
tentionally when talking to another person. One of the
and interaction techniques. Input devices are just the
best ways to avoid this problem is to use an implicit or
physical tools used to implement various interaction
invisible push-to-talk scheme (LaViola, 1999). A push-
techniques. In general, many different interaction tech-
to-talk interface lets users tell the application when they
niques can be mapped onto a given input device. The
are speaking to it. In order to maintain the naturalism of
question is how naturally, efŽciently, and appropriately a
the speech interface, one can embed the “push” into
given input device will work with a given technique.
existing interaction techniques so the user does not have
Input devices are also governed by the degrees of free-
the burden of remembering the signal to the application
dom (DOFs) they have. In general, an input device with
that a voice command is about to be issued.
a smaller number of DOFs can be used to emulate a
device with more DOFs with the addition of buttons or
modiŽer keys.
3 Interaction Techniques
Input devices can be roughly categorized based on
the types of events they generate. Devices that generate 3.1 Navigation
one event at a time based on the user are considered
The task of navigation is the most prevalent user
discrete-input devices. A button press or other action
action in most large-scale 3-D environments, and it pre-
signals an event, which is usually a Boolean (up/down)
sents challenges such as supporting spatial awareness,
value. Pinch gloves, developed by Fakespace, are an ex-
providing efŽcient and comfortable movement between
ample of a discrete-input device (the user pinches two
distant locations, and making navigation lightweight so
or more Žngers to signal an event). In contrast to dis-
that users can focus on more-important tasks. We subdi-
crete-input devices, continuous-input devices generate a
vide navigation into the motor component called travel
stream of events. Two of the most common continu-
and the cognitive component called wayŽnding.
ous-input devices are position/orientation trackers and
Navigation tasks can generally be classiŽed into three
datagloves, which transmit bend angles of the Žngers. categories. Exploration is navigation with no explicit
Devices that combine both discrete and continuous target: the user is simply investigating the environment.
events to form single, more-exible devices are called Search tasks involve moving to a particular target loca-
combination or hybrid-input devices. Examples of hy- tion. Finally, maneuvering tasks are characterized by
brid devices include the Ring Mouse (a small device short-range, high-precision movement that are used to
worn on the user’s Žnger that combines ultrasonic place the viewpoint at a more advantageous location for
tracking with two buttons) and pen-based tablets, which performing a particular task.
are becoming more popular in VE applications because Travel is a conceptually simple task—the movement
they give users the ability to interact in two dimensions. of the viewpoint from one location to another. Further,
Speech input is unique because the “device” is the viewpoint orientation is usually handled in immersive
human voice. Speech provides a nice complement to VEs by head tracking, so only techniques for setting
other input devices and, as a result, it is a natural way to viewpoint position need be considered.
combine different modes of input (multimodal interac- There are Žve common metaphors for travel interac-
tion). In general, speech input can be a valuable tool in tion techniques, and most published interaction tech-
3-D user interfaces, especially when both the user’s niques for travel Žt into one of these Žve categories:
hands are occupied. A major myth of speech recognition
is that having a good speech recognizer will solve all c Physical movement: The user’s body motion is
problems. Unfortunately, many other issues need to be used to travel through the environment. Examples
considered when dealing with speech input. Letting the include wide-area motion tracking, walking in
 Bowman et al. 99

Figure 1. Example travel techniques. Virtual Motion Controller, a physical movement travel technique (left). Placing markers on a map in a
route-planning technique (right).

place, and locomotion devices (Žgure 1, left) such travel metaphor and includes techniques such as
as treadmills or stationary bicycles. Such techniques gaze-directed steering (wherein the user’s head ori-
are appropriate when an enhanced sense of presence entation determines the direction of travel) or
is required or when the application requires the user pointing (in which hand orientation is used). Steer-
to experience physical exertion when traveling. ing techniques are general and efŽcient.
c Manual viewpoint manipulation: The user’s hand c Target-based travel: The user speciŽes the destina-
motions are used to effect travel. For example, the tion, and the system handles the actual movement.
user “grabs the air” and pulls himself along as if This may take the form of “teleportation,” in which
with a virtual rope. Another type of technique uses the user jumps immediately to the new location, or,
a selected object as a center point around which preferably, the system may perform some transi-
user motion can be speciŽed. These techniques can tional movement between the starting point and
be efŽcient and easy to learn, but they can also the destination. Target-based techniques are very
cause fatigue. simple from the user’s point of view.
c Steering: Steering is the continuous speciŽcation of c Route planning: The user speciŽes the path that
the direction of motion. This is the most common should be taken through the environment, and the
100 PRESENCE: VOLUME 10, NUMBER 1

system handles the actual movement. The user may centered and environment-centered support. User-
manipulate icons or draw a path on a map of the centered support includes factors such as a large Želd of
space or in the actual environment in order to plan view, the inclusion of visual motion and vestibular (real-
a route (Žgure 1, right). These techniques allow the motion) cues, and nonvisual support such as audio.
user to control travel while he retains the ability to These cues may have a substantial impact on the
perform other tasks during motion. sense of presence, but this requires further research.
Environment-centered support can be partitioned into
Besides choosing a metaphor, other design issues for structural organization and cues. Structural organization
travel techniques include velocity control and the use of deŽnes how clearly different parts of the environments
constraints or guides to aid travel. Bowman et al. have can be identiŽed and related to other parts. A good ex-
performed a variety of experimental evaluations (Bow- ample of structuring is the set of legibility techniques
man, Davis, Badre, & Hodges, 1999; Bowman & (Ingram & Benford, 1995). Cues consist of real-world
Hodges, 1997; Bowman, Johnson, & Hodges, 1999; wayŽnding principles that are transferred to a VE. The
Bowman, Koller, & Hodges, 1997), resulting in quanti- most commonly applied cues are artiŽcial, such as maps,
tative comparisons of techniques and serving as the basis compasses, and grids. This category could also include
for design guidelines. For example, designers should architectural cues like lighting, color, and texture,
consider whether travel is used as an end unto itself or and natural environment cues like a horizon and
in the service of some other task. (Target-based tech- atmospheric perspective.
niques, because of their simplicity, will allow the user to Studies with maps (Darken & Cevik, 1999) in VEs
focus on the more relevant task.) Another important have shown some wayŽnding performance gains. Addi-
guideline is that user training and instruction may be as tional evaluations need to be performed to learn how
important as the technique used. Users with sophisti- the content of the environment inuences the applica-
cated strategies will remain more spatially oriented than tion of cues. Most cues have been applied to environ-
will those with naõ´ve travel strategies. ments that are strongly related to real-world environ-
WayŽnding, the counterpart of travel, can be de- ments. In any case, wayŽnding support needs to be
scribed as the cognitive process of deŽning a path carefully implemented to allow useful navigation
through an environment, thereby using and acquiring through an environment.
spatial knowledge to build up a cognitive map of an en-
vironment. Spatial knowledge consists of landmark, pro-
3.2 Selection and Manipulation
cedural, and survey knowledge (Thorndyke & Hayes-
Roth, 1982). The usage and acquisition of spatial Interaction techniques for 3-D manipulation in
knowledge is inuenced by such factors as the reference VEs should provide means to accomplish at least one of
frame (Žrst-person egocentric or third-person exocen- three basic tasks: object selection, object positioning,
tric) and the travel technique. and object rotation. Because direct hand manipulation is
Users should receive wayŽnding support during VE a major interaction modality not only in the 3-D virtual
travel. Individual users have a wide range of spatial abili- world but also in natural physical environments, the
ties, and the extra degrees of freedom within a VE can design of interaction techniques for object selection and
easily cause disorientation. In the case of VE training, manipulation has a profound effect on the quality of the
with the goal of transferring knowledge from the VE to entire VE user interface.
the real world, the application and environment should The classical approach to design manipulation tech-
be designed to support the transfer (using techniques niques is to provide the user with a “virtual” hand—a
discussed below); otherwise, the training session can 3-D cursor, often shaped like a human hand, whose
easily become counterproductive. movements correspond to the movements of the hand
WayŽnding support can be subdivided into user- tracker (Žgure 2, left). Selection and manipulation sim-
 Bowman et al. 101

Figure 2. “Classical” virtual hand technique (left) and nonlinear mapping function of Go-Go technique (right).

ply involve touching an object, then positioning and the spotlight, further disambiguation of the target ob-
orienting this virtual hand within the VE. The virtual ject is required. The aperture technique (Forsberg,
hand technique is rather intuitive because it simulates a Herndon, & Zeleznik, 1996) uses a conic pointer
real-world interaction with objects, but only those ob- whose direction is deŽned by the location of the user’s
jects within the area of reach can be picked up. eye (estimated from the head location) and a hand sen-
A number of techniques have been suggested to over- sor. The user can control the size of the selection vol-
come this problem. The Go-Go technique (Poupyrev, ume simply by bringing the hand sensor closer or mov-
Billinghurst, Weghorst, & Ichikawa, 1996) allows the ing it farther away (Žgure 3, right). The image plane
extension of the user’s reach by using a nonlinear map- family of interaction techniques (Pierce et al., 1997)
ping applied to the user’s hand extension. When the develops and extends this idea.
user extends the hand farther than a threshold distance All the techniques described above provide users with
D, the mapping becomes nonlinear and the virtual arm tools that allow them to select or reach further in the
“grows” (Žgure 2, right). Different mapping functions immersive virtual world. An alternative approach would
can be used to achieve different control-display gain be to allow the user to manipulate the relative scale of
between real and virtual hands (Bowman & Hodges, the virtual world. One of the earliest uses of this ap-
1997). proach was in the 3DM immersive modeler (Butter-
The other common way to select and manipulate ob- worth, Davidson, Hench, & Olano, 1992), in which
jects in VEs is to point at them using a virtual ray ema- users could “grow” or “shrink” themselves to manipu-
nating from the virtual hand. When the virtual ray inter- late objects of different sizes. The World-in-Miniature
sects an object, it can be picked up and manipulated (WIM) technique (Stoakley, Conway, & Pausch, 1995)
(Žgure 3, left). Several variations of ray casting have provides a handheld model of the VE. The user can
been developed to help users in selecting very small or then indirectly manipulate virtual objects by interacting
faraway objects. For example, the spotlight technique with their representations in the WIM.
(Liang & Green, 1994) provides a conic selection vol- Because all manipulation techniques have particular
ume, so that objects falling within the cone can be easily strengths and weaknesses, a number of attempts have
selected. However, when more than one object falls into been made to integrate and combine their best features.
102 PRESENCE: VOLUME 10, NUMBER 1

Figure 3. Ray-casting technique (left) and aperture techniques (right).

For example, the Virtual Tricorder (Wloka & Green- Thus, some similarities can be seen between system con-
Želd, 1995) combines ray casting for selection and ma- trol and object selection techniques.
nipulation with techniques for navigation and level-of- In desktop applications, the use of commands has
detail control within one universal tool. Other examples received much attention. Unfortunately, interaction
are the HOMER techniques (Bowman & Hodges, styles used in desktop environments, like pulldown
1997), world scale grab (Mine, Brooks, & Sequin, menus and command-line input, are not always usable
1997) and Voodoo Dolls (Pierce, Stearns, & Pausch, within a VE. One of the basic problems of VE system
1999). control is that a normally one- or two-dimensional task
The variety of reported interaction techniques can be becomes three-dimensional, which reduces the effective-
overwhelming for the developer. However, some gen- ness of traditional techniques. For example, touching a
eral principles regarding the choice of manipulation menu item oating in space is much more difŽcult than
techniques can be stated. None of the techniques can be selecting a menu item on the desktop, not only because
identiŽed as the “best”: their performance is task and the task has become 3-D, but also because the impor-
environment dependent. Often, nonrealistic techniques tant constraint of the physical desk on which the mouse
have better performance than those based on the real rests is missing. Evaluation results for system-control
world. Finally, it is important to implement constraints techniques are relatively sparse. Although many ad hoc
and limit DOFs if possible. Evaluation of virtual manip- implementations have been reported, system control has
ulation techniques helps to quantify performance and is not been studied in a structured way.
an important ongoing research area. We can categorize system-control techniques for im-
mersive VEs into four groups, namely graphical menus
(visual representations of commands), voice commands
3.3 System Control
(menus accessed via voice), gestural interaction (com-
System control refers to a task in which a com- mand sets accessed via gesture), and tools (virtual ob-
mand is applied to change either the state of the system jects with an implicit function or mode). Also, hybrid
or the mode of interaction. The issuing of a command techniques exist that combine several of the types.
always includes the selection of an element from a set. System control is often integrated within another uni-
 Bowman et al. 103

versal interaction task. Due to this integration, one tioned previously, most efŽcient selection techniques are
should avoid disturbing the ow of action of the main essentially 2-D, although further manipulation may re-
interaction task. The user should stay focused on the quire a 3-D interaction technique. By taking advantage
task. “Modeless” interaction (in which the mode of the beneŽts of both 2-D and 3-D interaction tech-
changes are very natural) is ideal. One way of support- niques, we can create interfaces for 3-D applications
ing easy access to a system-control interface is to use a that are easier to use and more intuitive for the user.
natural spatial reference, such as a Žxed position relative Seamless integration of 2-D and 3-D interaction tech-
to the user’s head or body, for placement of the inter- niques is a critical design consideration from both a
face. This guideline is mostly applicable to graphical physical and logical perspective. Physical integration is
menus, but tools also beneŽt from a strong spatial refer- important because we do not want to make it difŽcult
ence. Another method to allow a more seamless integra- for users to switch between 2-D and 3-D devices. Logi-
tion of system control into a ow of action is to use a cal integration is also important because we want the
multimodal system control interface. devices in the application to know whether they are
After accessing a system-control interface, one has to used for 2-D or 3-D interaction. This contextual appli-
select a command. When the set of functions is large, cation-based information helps to reduce the user’s cog-
one needs to structure the items. This might be
nitive load.
achieved by methods like context-sensitive menus, or by
The 2-D/3-D interfaces can be roughly classiŽed into
clearly communicating the hierarchy of items and
three categories. Note that, in all categories, some type
(sub)menus.
of physical surface is required for 2-D input. The distin-
Finally, the designer should try to prevent mode er-
guishing characteristic of these interfaces is how the
rors by providing the user with appropriate feedback
physical 2-D surfaces are utilized. The Žrst category
during and after selection of a command. Mode errors
covers applications that use fully immersive displays such
can be highly disruptive to the ow of action in an ap-
as HMDs, where the user cannot physically see the 2-D
plication.
surface. Here, the 2-D surface is usually a piece of
As in other areas, evaluations are needed which test
tracked plastic or pen-based tablet, and users must have
different system-control techniques within realistic ap-
a graphical representation of the surface in order to in-
plication domains. Further research is needed to deter-
teract with it in the virtual world. Examples of this type
mine the performance of these techniques and their ap-
plicability to different application situations. are the Virtual Notepad, a system for writing and anno-
tating in VR (Poupyrev, Tomokazu, & Weghorst,
1998) and the Virtual Habitat (Bowman, Wineman,
4 Two-Dimensional Interaction in Three- Hodges, & Allison, 1998).
Dimensional Environments The second category of 2-D/3-D interfaces cover
applications that use semi-immersive displays such as
A common misconception of 3-D user interface workbenches. The physical 2-D interaction surface is
design is that, because the applications usually contain usually either on top of the workbench display so users
3-D worlds in which users can create, select, and manip- can directly interact with the display surface, or on a
ulate 3-D objects, the interaction design space should tracked, transparent tablet that users can hold in their
utilize only 3-D interaction. In reality, 2-D interaction hand. In the latter case, graphics are projected on the
offers a number of distinct advantages over 3-D interac- primary display but virtually appear as if they are on the
tion techniques for certain tasks. If haptic or tactile dis- surface of the tablet. Examples of this category are the
plays are not present, 2-D interaction on a physical sur- ErgoDesk system (Forsberg et al., 1998), a modeling
face provides a sense of feedback that is especially useful application using the physical display surface for a 2-D
for creating objects, writing, and annotating. As men- interface, and the Transparent Pad, a clear tablet that
104 PRESENCE: VOLUME 10, NUMBER 1

users can hold to interact with a landscaping design ap- Designing 3-D interfaces has not yet reached the
plication (Schmalsteig, Encarnacao, & Szalzvari, 1999). same state of maturity. There are still no cohesive 3-D
The third category uses separate 2-D display surfaces, interface paradigms similar to the WIMP paradigm in
such as handheld computers and pen-based LCD tab- desktop user interfaces. Although many 3-D interaction
lets. An example of this type of interface is the use of a techniques have been reported in the literature and
Palm Pilot in a CAVE-like device for camera, environ- many human-factors issues of multidimensional interac-
ment, and geometry controls (Watsen, Darken, & tion have been investigated, there is still no common
Capps, 1999). In this area, there are many open re- vision on how these research results can be combined
search issues to explore, especially when dealing with together to form a cohesive picture that would guide
wireless technology and the weight of these devices. the interface designer in designing and building spatial
interfaces. One of the reasons for this is that the design
space in 3-D user interfaces is signiŽcantly larger than in
5 Philosophies of 3-D Interaction Design 2-D, and large portions of it remain unexplored. De-
signers have to deal with a large variety of input and
The techniques and guidelines presented in this output devices and technology that is still rapidly devel-
paper come from years of work by many researchers in oping: new displays and sensors that require innovation
the Želd. Often, there is a debate as to the most effec- of appropriate interaction techniques and consequent
tive methodology or philosophy for the development of reevaluation of accumulated design experience and
good 3-D user interfaces. In this section, we compare knowledge.
and contrast an “artistic” philosophy of design with a How then can developers approach 3-D interface de-
systematic philosophy. In our view, these two strategies sign? The successful 3-D interface design should be
complement one another: a ash of insight (art) can based, Žrst, on existing research on human factors in
lead to good systematic research, and vice versa. Both computing systems. Second, it should reuse interaction
approaches can lead to novel and effective techniques, techniques and ideas developed by researchers. Third, it
and principles and guidelines for design. has to employ creativity and simple approaches that can
help to invent interfaces and new interaction tech-
niques. Finally, it must use existing design models and
5.1 Artistic Philosophy
strategies of 3-D interface design.
Shneiderman (1998) identiŽed three “pillars of Examples of basic human-factors principles that can
successful user interface design”: guideline documents, be directly used in 3-D interface design include research
user interface software tools, and expert review and on two-handed interaction (for example, Hinckley,
usability testing. These pillars specify major sources of Pausch, ProŽtt, Patten, and Kassell (1997)), constraints,
guided design decisions, such as the Macintosh Human sensory feedback, and multimodal interaction, as well as
Interface Guidelines or the interface API provided as basic interface-design guidelines that are applicable from
part of the Macintosh operating system. Together they 2-D interface design: the requirements of simplicity,
outline and implement the very basic elements of the consistency, error prevention, and so on.
desktop user interface, deŽne their functionality, pur- Techniques for inventing 3-D interfaces can be
pose, appropriate visual appearance and so forth. These broadly categorized as those borrowing from the real
and other guidelines and tools provide designers not world and those based on “magic.” Examples of realistic
only with basic interface building blocks, liberating techniques include the exact simulation of the physical
them from the necessity to invent and implement them interaction between humans and environment, borrow-
themselves, but also with a clear vision of how these ing from other areas of human activities (such as movies
interface blocks Žt together and can be used to design and architecture), adapting real world tools in the form
user interfaces for a particular product. of 3-D widgets, and borrowing design ideas from 2-D
 Bowman et al. 105

user interfaces. The magic approach involves designing ability evaluation of VE applications must also be
interaction techniques around cultural clichés and meta- used (Hix et al., 1999).
phors—the ying carpet and magic wand metaphors—as
well as conscious violation of user assumptions (Pierce,
personal communication, 2000). 6 Applications
Although these techniques allow us to design only
short-term, and often ad hoc, solutions, they do provide To illustrate the practical usage of some of the
a foundation for the scientiŽc scrutiny that can analyze, techniques, devices, and concepts described in the ear-
categorize, and evaluate proposed ideas and techniques, lier sections, we present some application case studies in
giving rise to the systematic approach to interface design this section.
as discussed below.

6.1 Exploring Complex Data

5.2 Systematic Philosophy Visualization s with the Cubic Mouse

The systematic approach to interaction design is The use of complex visualization systems is wide-
characterized by a study of user tasks, existing interac- spread in application domains such as the automobile
tion techniques, and characteristics of the user, environ- industry, the medical Želd, and the oil and gas industry.
ment, or system that might affect performance. In gen- These visualization systems need to handle massive
eral, this approach is slow and methodical, with amounts of data, use appropriate visualization methods,
incremental improvements in performance rather than and apply effective interaction techniques to inspect and
sudden leaps. manipulate data. Traditional input devices often form a
An important component of the systematic philoso- bottleneck when interacting with such systems, due to
phy is classiŽcation, usually done in the form of taxono- their imprecision and need for regular mode changes. At
mies. By classifying, categorizing, and decomposing the German National Research Center for Information
tasks and techniques, we gain a much deeper under- Technology (GMD), research in visualization has led to
standing of them and provide ourselves with a frame- a new input device called the Cubic Mouse (Froehlich
work for the design of new techniques and the evalua- & Plate, 2000).
tion of existing ones. One type of taxonomy, based on a The Cubic Mouse (Žgure 4) is a cube-shaped de-
hierarchic decomposition of a task and then a listing of vice with three rods going through the primary axes.
technique components for the lowest-level subtasks, Within the box, a magnetic sensor is placed to track
lends itself to guided design, a way to use the taxonomy orientation and position. The Cubic Mouse supports
for the generation of new interaction technique designs. highly precise two-handed interaction with complex
This approach can even be implemented in software so data sets. Generally, the entire device is used to create
that new interaction techniques can be prototyped and an appropriate viewpoint on a scene, while the rods
tested extremely quickly. are used to translate and rotate objects within the
A second crucial aspect of the systematic approach scene in a constrained way. During several informal and
is evaluation. Although many think of design and formal evaluations, users showed increased precision in
evaluation as separate concepts, they are in fact comparison to two gloves or a stylus, and clearly preferred
tightly connected. The iterative design process used the Cubic Mouse for precise manipulations.
in standard HCI contexts applies to VEs as well.
Evaluation of a design results in changes to that de-
6.2 Multimodal Interfaces in VEs
sign, which can again be evaluated. In the area of 3-D
interaction, basic evaluation of interaction techniques Multimodal interaction can be deŽned as the
is still important, but formative and summative us- combination of multiple input and output modalities
106 PRESENCE: VOLUME 10, NUMBER 1

for remembering important viewpoints and recording

animations. These types of interactions present a more
natural interface to the user.

6.3 VEs for Design Education

Architectural design is often touted as an applica-

tion area for which virtual environments should be
ideal. Beyond walkthroughs and validation environ-
ments, however, there are few applications of immersive
design. The Virtual Habitat (Bowman et al., 1998) is a
constrained design space focused on education, rather
than a generic design environment aimed at professionals.
The goal of the Virtual Habitat is to allow students to
apply the principles of environmental design. Students
Figure 4. The Cubic Mouse (copyright B. Froehlich, GMD). are immersed in a virtual zoo exhibit and use interactive
design tools to make modiŽcations to the terrain, the
locations of rocks and trees, and the location and view-
ing angle of the visitor viewpoints. The task of immer-
to provide users with a richer set of interactions com- sive design is made possible because students do not
pared to traditional unimodal interfaces. Output mo- have to start with a blank environment and because the
dalities can be combined to allow for sensory substi- design tools are constrained by the speciŽc tasks of ex-
tution and parallel processing of information. There hibit design and by the principles of good environmen-
are several basic ways to combine input modalities tal design.
(Martin, 1998), including complementarity (two or The Virtual Habitat exhibits complex interaction. Us-
more input modalities complement each other when ers must navigate, select and manipulate objects, and
they combine to issue a single command), concur- control the display of information. The system supports
rency (two or more input modalities are concurrent different types of tasks and user preferences by provid-
when they issue different commands that overlap in ing at least two ways to do most tasks: a direct tech-
time), and transfer (two input modalities transfer in- nique in which the action takes place in the environ-
formation when one receives information from another ment, and an indirect technique that uses a 2-D
and uses this information to complete a given task). interface displayed on a tablet. The interaction tech-
These multimodal input styles have been imple- niques were chosen based on the results of experiments
mented in the context of VR-based scientiŽc visualiza- comparing various travel, selection, and manipulation
tion applications. In particular, the Multimodal Scien- techniques. A usability study has shown this application
tiŽc Visualization Tool (MSVT) allows users to view to perform very well in its intended domain. First-time
and interact with uid ow about a data set by utilizing users of the system all completed novel designs during
hand gesture and voice input (LaViola, 2000). MSVT sessions of one hour or less.
uses input combination styles such as complementarity
and transfer for the creation, manipulation, and removal
of visualization tools. Users can ask the application for a 7 Conclusions
particular tool while showing it where to place the tool.
In addition, users can concurrently manipulate the data Three-dimensional interaction research will only
set and visualization tools while issuing voice commands continue to grow in importance as VEs become more
 Bowman et al. 107

complex and widespread. We have given a brief intro- metaphors, and applications. We stress that researchers
duction to the Želd and some of the issues involved in must continue to perform evaluations of usability and
designing effective 3-D user interfaces. For further in- performance, and that further guidelines for the devel-
formation, readers should consult the references in the opment of effective 3-D interfaces are necessary. Such
3DUI bibliography (Poupyrev & Kruijff, 2000) or join research has a direct impact on the acceptance of VEs as
(by emailing the authors) the 3DUI mailing list, which real-world tools, and on the levels of user satisfaction
is devoted to the discussion of this research area. and productivity in VE applications.
The principles of traditional user interface design can
often be applied directly to 3-D user interfaces but 3-D
interaction has its own unique characteristics that re- References
quire new design guidelines, some of which have been
discussed above. Several generic principles for 3-D user Begault, D. (1994). 3D Sound for Virtual Reality and Multi-
interface design can be stated based on the current state media. Academic Press.
of the art in this area. These include: Bowman, D., Davis, E., Badre, A., & Hodges, L. (1999).
Maintaining spatial orientation during travel in an immer-
c Consider “magic” interfaces in place of “natural” sive virtual environment. Presence: Teleoperators and Virtual
ones when tasks require productivity and efŽciency. Environments, 8(6), 618 – 631.
Natural interaction should be used when a replica- Bowman, D., & Hodges, L. (1997). An evaluation of tech-
tion of the physical world is important. niques for grabbing and manipulating remote objects in
c Choose interaction techniques based on the re- immersive virtual environments. Proceedings of the ACM
quirements of the application. The same set of tech- Symposium on Interactive 3D Graphics, 35–38.
niques will not work well in every situation. Bowman, D., Johnson, D., & Hodges, L. (1999). Testbed
c Limit the required degrees of freedom for input evaluation of VE interaction techniques. Proceedings of the
whenever possible, and provide physical or virtual ACM Symposium on Virtual Reality Software and Technol-
constraints to help guide user input. ogy, 26 –33.
c Take advantage of the increased exibility afforded Bowman, D., Wineman, J., Hodges, L., & Allison, D. (1998).
Designing animal habitats within an immersive VE. IEEE
by whole-body input, multimodal I/O, and novel
Computer Graphics & Applications, 18(5), 9 –13.
input devices, but also give the user structure and
Bowman, D. A., Koller, D., & Hodges, L. F. (1997). Travel
support in managing the increased complexity.
in immersive virtual environments: An evaluation of view-
There are still many unanswered questions and re- point motion control techniques. Proceedings of the Virtual
search issues in the Želd of 3-D interaction. Some of the Reality Annual International Symposium, 45–52.
most important unresolved issues are: Burdea, G. (1996). Force and Touch Feedback for Virtual Re-
ality. Wiley Interscience.
c Will a standard for 3-D interfaces ever be practical? Butterworth, J., Davidson, A., Hench, S., & Olano, M.
c How must interaction change between different (1992). 3DM: A three dimensional modeler using a head-
display devices (for example, HMD and CAVE)? mounted display. Proceedings of the ACM Symposium on In-
c How can VEs best support complex system-control teractive 3D Graphics, 135–138.
tasks (such as very large sets of menu items)? Darken, R., & Cevik, H. (1999). Map usage in virtual envi-
c How can we determine which 3-D interaction tech- ronments: Orientation issues. Proceedings of IEEE Virtual
niques work best for a given application? Reality, 133–140.
c How do various 3-D interaction styles affect the Forsberg, A., Herndon, K., & Zeleznik, R. (1996). Aperture
user’s sense of presence? based selection for immersive virtual environments. Proceed-
ings of the ACM Symposium on User Interface Software and
We hope that this overview helps to encourage fur- Technology, 95–96.
ther work on 3-D interaction techniques, user interface Forsberg, A., LaViola, J., & Zeleznik, R. (1998). ErgoDesk: A
108 PRESENCE: VOLUME 10, NUMBER 1

framework for two and three dimensional interaction at the Pierce, J., Stearns, B., & Pausch, R. (1999). Voodoo Dolls:
ActiveDesk. Proceedings of the Second International Immer- Seamless interaction at multiple scales in virtual environ-
sive Projection Technology Workshop. ments. Proceedings of the ACM Symposium on Interactive 3D
Froehlich, B., & Plate, J. (2000). The Cubic Mouse: A new Graphics, 141–146.
device for three-dimensional input. Proceedings of ACM Poupyrev, I., Billinghurst, M., Weghorst, S., & Ichikawa, T.
CHI. (1996). The Go-Go interaction technique: Non-linear map-
Herndon, K., van Dam, A., & Gleicher, M. (1994). The chal- ping for direct manipulation in VR. Proceedings of the ACM
lenges of 3D interaction. SIGCHI Bulletin, 26(4), 36 – 43. Symposium on User Interface Software and Technology, 79 –
Hinckley, K., Pausch, R., ProŽtt, D., Patten, J., & Kassell, N. 80.
(1997). Cooperative bimanual action. Proceedings of CHI: Poupyrev, I., & Kruijff, E. (2000). 20t h Century 3DUI Bibli-
Human Factors in Computing Systems, 27–34. ography: Annotated Bibliography of 3D User Interfaces of the
Hix, D., Swan, J., Gabbard, J., McGee, M., Durbin, J., & 20t h Century. Available at: http://www.mic.atr.co.jp/
King, T. (1999). User-centered design and evaluation of a , poup/3dui/3duibib.htm.
real-time battleŽeld visualization virtual environment. Pro- Poupyrev, I., Tomokazu, N., & Weghorst, S. (1998). Virtual
ceedings of IEEE Virtual Reality, 96 –103. notepad: Handwriting in immersive VR. Proceedings of the
Ingram, R., & Benford, S. (1995). Legibility enhancement for Virtual Reality Annual International Symposium, 126 –132.
information visualization. Proceedings of IEEE Visualization, Schmalsteig, D., Encarnacao, L., & Szalzvari, Z. (1999). Us-
209 –216. ing transparent props for interaction with the virtual table.
LaViola, J. (1999). Whole-hand and speech input in virtual Proceedings of the ACM Symposium on Interactive 3D Graph-
environments. Unpublished master’s thesis, Department of ics, 147–154.
Computer Science, Brown University, CS-99-15. Shneiderman, B. (1998). Designing the User Interface (3rd
LaViola, J. (2000). MSVT: A multimodal scientiŽc visualiza- ed.). Reading, MA: Addison Wesley Longman.
tion tool. Proceedings of the Third IASTED International Stoakley, R., Conway, M., & Pausch, R. (1995). Virtual real-
Conference on Computer Graphics and Imaging (forth- ity on a WIM: Interactive worlds in miniature. Proceedings
coming). of CHI: Human Factors in Computing Systems, 265–272.
Liang, J., & Green, M. (1994). JDCAD: A highly interactive Thorndyke, P., & Hayes-Roth, B. (1982). Differences in spa-
3D modeling system. Computer & Graphics, 4(18), 499 – tial knowledge obtained from maps and navigation. Cogni-
506. tive Psychology, 141982, 560 –589.
Martin, J. (1998). TYCOON: Theoretical framework and Watsen, K., Darken, R., & Capps, M. (1999). A handheld
software tools for multimodal interfaces. In John Lee (Ed.), computer as an interaction device to a virtual environment.
Intelligence and Multimodality in Multimedia Interfaces. Proceedings of the Third Immersive Projection Technology
AAAI Press. Workshop.
Mine, M., Brooks, F., & Sequin, C. (1997). Moving objects Wloka, M., & GreenŽeld, E. (1995). The virtual tricorder: A
in space: Exploiting proprioception in virtual environment uniŽed interface for virtual reality. Proceedings of the ACM
interaction. Proceedings of ACM SIGGRAPH, 19 –26. Symposium on User Interface Software and Technology, 39 –
Pierce, J. (2000). Personal communication. 40.
Pierce, J., Forsberg, A., Conway, M., Hong, S., Zeleznik, R., Youngblut, C., Johnson, R., Nash, S., Wienclaw, R., & Will,
& Mine, M. (1997). Image plane interaction techniques in C. (1996). Review of virtual environment interface technol-
3D immersive environments. Proceedings of the ACM Sympo- ogy (IDA Paper P-3186, Log: H96-001239). Institute for
sium on Interactive 3D Graphics, 39 – 44. Defense Analysis.