Consciousness Reframed III

Sensual Communication in Hybrid Reality

Maja Kuzmanovic

Summary
This paper focuses on the potential applications of hybrid reality in the cultural sphere,
where media and technologies can be used for multisensory stimulation and interaction. We
investigate the capacity of hybrid spaces to incite alternative states of consciousness, similar
to what mystics, alchemists and shamans describe as a journey towards rapture (a state of
overwhelming emotion). The two projects that we discuss here, GoTo and T-Garden,
explore the notion of sensual communication that occurs when the virtual and the physical
intermingle in the same reality. The media become tangible and the senses distributed.
Distancing ourselves from the modernist ideals of an independent mind, we strive towards
embodied, actualised imaginaries, through which people navigate following a corporeal
language.

Keywords
language, hybrid reality, tactility, transmutation, mysticism, alchemy, shamanism

Unity, Transmutation and Rapture

"Among medieval philosophers love rose from the soul toward the superior intelligencies. In
our humanist system it is by their passage through virtual worlds, by acquiring an angelic
body, that souls can best imagine humanity, from which perhaps follows, along with the
desire of learning, the growth of friendship among men." (Levi, 1997)

The theme of this paper is the link between networked communication, responsive media and
bodily sensations. It was inspired by our study of alternative (virtual) realities spanning 3000
years of European history. The 'virtual' in these worlds generally points to the spiritual, or in
a broader sense, to the potential worlds, alternative realities beyond ordinary consciousness.
The body was used as a vessel to transport the mind into the realm beyond the semantic,
'normalised' logic of the increasingly dogmatic and regulated societies. For the creators and
'inhabitants' of these realities, such as mystics, shamans and alchemists, the virtual realms
were channels for communication with a remote, immaterial entity: their god(s). Many
similarities between these ancient views and today's multiple-reality research can be found. A
few examples:

1. 'Spiritual realities' were exclusive domains of secret sects, 'the chosen ones', just as
today's VR technology too often remains in the hands of a few wealthy institutions, and it is
seldom accessible for experiment, that is certainly needed for the understanding of its full
potential. This technology offers the possibilities to extend our senses, and bridge the gap
between the immaterial world of thought and the sensual world of bodily experience.

2. The practitioners were constructing and exploring their individual identities through
alternative ways of intercourse (with God). To quote the Critical Art Ensemble,

"In a truly homeopathic reversal, the body was reformatted as a site for autonomy. The flesh
was explored as a means to freedom through sensual presence - female mystics physically
embraced god in the ensouled flesh of His decaying creations by tasting the wounds of lepers
and the vomit of the sick, and in feeling the pain of their own emaciated bodies. Decaying
flesh was transubstantiated in the holly fire of the mystic's desire to independently commune
with God." (CAE 1997)

3. They referenced the known models of the world, as metaphors to describe something
imageless and virtual. According to Turner (Turner 1974), these metaphors or symbols
'stretch between two poles: the ideological and the sensory. The ritual symbol join these two
opposing poles together and offers both emotion and social values."

4. The structure of their worlds is exploratory, based on their itinerant journeys towards
God(s). As Teresa of Avila describes her journey into the soul:

"These mansions are not arranged in a row one behind another, but variously -- some above,
others below, others at each side: and in the center and midst of them all is the chiefiest
mansion, where the most secret things pass between God and the soul." (St Teresa of Avila)

5. Shamans, alchemists and mystics search for unity, interconnectedness and spiritual
ecstasy; while at all times conscious that they are deeply rooted in their physical bodies, in
flesh. "The essence of everything is relation", a sentence by Meister Eckhart describes
(Davies, 1994) the underlying thought of many mystics and alchemists, such as Hildegard of
Bingen, Thomas Aquinas and Paracelsus. Having the body as a vessel of the intercourse, they
explore their immaterial worlds in the same way we explore hybrid realities today. They are
approached as uncharted territories, that encourage exploration, affecting not only their
communication abilities, but also their physical experience.

6. They use reality as a reference point, from which they set out on a vague journey
towards a vague goal, during which they both construct the virtual world and draw its maps,
turning the virtuality inside out, making it a part of their everyday lives. These topological
maps are their stories and songs.

The languages that the mystics, shamans and alchemists use to transfer information about the
experiences within the immaterial worlds have a unique sensual component. They are
opposed to documentation of facts, and opposed to describing laws and moral messages that
the audience/readers should follow. Instead, the fluidity of imagery and experience dissolves
all differences between gendered structures and inspires participation.

These tongues speak in nuances and atmospheres, the content grows through subtle
transformations in the environment. Liquid structure of these worlds allows change from one
participant to the other, it can incorporate both a fact and its opposite and still seem
consistent. Using a language that addresses multiple senses means that the content spirals
from one state to another, targeting to inspire participants' material experience, while leaving
plenty of room for mental association.

GoTo

GoTo is research project exploring novel ways of storytelling in Virtual Reality (CAVE). It
is a system of associatively and contextually linked stories, sharing 15 common themes, such
as pain, rhythm, change, interiority etc. The timeline of the narrative is generated
dynamically, based on the participants' path through the themes, that depends on their
movement in the CAVE. Both the interface and the interaction are kept as simple and
transparent as possible, in order not to disrupt the feeling of immersion, but still give enough
subtle clues for navigation.
As the VR environment is continuous, so should be the connections between all sensory
experiences. These experiences should be closely linked to the passages between the different
content parts: the hyperlinking in VR needs the development of sensually satisfying
transitions. The transitions explore the insides of a link, and explicitly show the participants
the impact of their choices on the environment.

Very active users mould the environment as clay, and create a very personal view of the
StoryScape. At some point, the predefined chunks of content could warp into an endless knot
of transitions. It is possible that being immersed in such an environment might make the
users motion sick. However, this might also mean that we are developing a new physical
sense, that will make the language of virtual reality understandable for our bodily senses, as
happened with montage in cinema, that in the beginning made the audience confused and
sometimes even nauseous. This VR sensation is a feeling of being in constant mutation,
without a progress or a goal, other than to mutate again.

The only interface elements are the barely visible swarming particles, that appear as
suggestions of possible paths. The particles are superimposed on a texture of changing
symbols.

Image 1: GoTo Interface

The participants' movements across this texture are tracked, and influence the path through the
stories. By turning their head, the users trigger events within one particular story fragment.
For the interaction in GoTo we used a metaphor of a daydream: the participants are casually
passing through the environment, while the media elements, such as video textures and
sounds are projected from different directions. They do not disrupt the environment, as they
are submerged under a semi-opaque shell.

This can be compared to the mirrored images seen in a polished shop window, or incidental
fragments of conversations in a noisy street. By moving their head and focusing on a specific
media element, the participants 'call' it to the surface and are able to experience its contents.
This also triggers the changes in the overall atmosphere of the story, as more contextually
linked media elements might surface, pulled by the 'magnetic' force of the chosen content.
Other, less contextually imminent media will disappear in the deeper layers of the
environment.

Image 2: Media Selection

The GoTo0 environment should be experienced as a very fragile system. It consists of a
meters thick structure, made of thin layers of an ice or glass like substance. With every step,
one falls deeper through the structure, that breaks open in big chunks at first, but the
'vibration' caused by the step, will break the big chunks into smaller fragments, so that the
surface becomes increasingly less flat. When we map a video texture on the big portions of
the environment, they are fragmented and mirrored in multiple perspectives.

Visually speaking, the storySpace is undifferentiated at first. As the participants move, they
'crack' the environment, as walking on ice. Depending on how much time has passed, and
how much of the environment is formed, the more or the less media textures the participants
will perceive visually: in the beginning, there are only sound and tactile feedback. The more
the environment opens up, the more visual the experience becomes:

The environment should be experienced as a very fragile system. It consists of a meters thick
structure, made of thin layers of an ice or glass like substance. With every step, one falls
deeper through the structure, that breaks open in big chunks at first, but the 'vibration' caused
by the step, will break the big chunks into smaller fragments, so that the surface becomes
increasingly less flat. When we map a video texture on the big portions of the environment,
they are fragmented and mirrored in multiple perspectives.

T-Garden

T-Garden is a responsive environment where visitors can put on sound, dance with images
and play with media together in a tangible way, constructing musical and visual worlds 'on
the fly'. The performance dissolves the lines between performer and spectator by creating a
social, computational and media architecture that allows the visitor-players to sculpt and
shape the overall environment.

Image 3: Excerpt from T-Garden Concept

All media (clothing, image, sound) in the T-Garden environment follow one central theme:
transmutation. Within this theme, the media will explore the connections between different
mutating systems, such as alchemy, ecology, memory, archaeology and recognition. Melodic
and rhythmic flows and cycles, morphs, transformations and pliability are some of the
characteristics of media that will be explored and developed. Aural and visual density can be
influenced and guided depending on the play of visitors. The entire space should appear
experientially as if it was shapeable and responsive in a fluid and choreographic manner-where
the visitors own bodies can meld inside an alchemical landscape and, like the principal goal
of alchemy itself, result in a transformation not only of the media but the visitors themselves.

The alchemy of T-Garden is a self-creating activity that does not accept the notion of a static
world. An alchemical reality is a world in constant development-in becoming. By
transforming the media around them, the visitors perform an alchemical journey where
physical and virtual (or spiritual), exterior and interior, micro and macro exist in "the
collision of two elastic spheres." The image becomes tangible, the sound malleable and the
clothing ethereal. The media use a dynamic language that can be compared to the movement
of verbs instead of the symbolism of nouns. This is a language that connects the sensual
experience of the visitors with the processes of memory and sedimentation. T-Garden should
be experienced as an alchemical world, whose matter is not actually solid, but merely a stress,
a strain in the field of time and space. It is the visitors gestures ((that are not so different
from everyday gestures of touching, brushing along other bodies, moving and falling, etc.)
that define the matter of T-Garden. For without the human gestures, it would remain in a
chaotic flux.

As visitors enter the performance, they find an array of clothing from which they can choose
to don. The clothing has specific exaggerated physical qualities of, for example, weight, size
and material. This clothing is embedded with wearable sensing devices as well as small audio
speakers. Individually, the visitors enter into several private vestibules-rehearsal studios where
they can play with streams of sounds and compositional effects that is produced by and
played within the clothing. There the visitors can reveal the aural and physical properties of
their garment instrument, and gradually learn how to modulate and change the sounds they
are receiving.

After practicing, the players then enter a circular room, thick with sound and image. The floor
is covered with transforming, polymorphous video and computer- generated textures: organic
forms, elemental and microscopic liquid and solid state changes. These phantasmagoric
textures appear to breathe and dance according to the sound patterns in the room. In this
garden, as the visitors pass near each other, their clothes will appear to howl and squeal -
patterns of sound bleeding from one body to another. As the visitors move about, their
locations and groupings will strengthen and lighten the density of the visual environment
while varying the melodic and rhythmic aspects of the sound space. Memory, population
density and bodily proximity affect the dynamics of the room, causing growth, decay,
infection and contamination in the visual environment.

Visitors leave traces and "converse" with each other in musical and visual phrases as they
weave their way through the room. The traditional roles of spectator and actor dissolve into a
field of performance as gesture immanent in ordinary life, where social play emerges without
explicit rules or grammar.

Image 4: T-Garden Visual Media Sample

References

Critical Art Ensemble. 1997. Flesh machines. New York: Autonomedia

Davies, O. Ed. 1994. Meister Eckhart: Selected Writings. London: Penguin Classics
Levi, P. 1997. Collective Intelligence. Cambridge, Massachusetts: Perseus Books
Turner, V. 1974. Dramas, fields and metaphors: Symbolic action in human society. Ithaca:
Cornell University Press.
St. Teresa of Avila. Interior Castle. http://www.ccel.org/t/teresa/castle/castle.html

Maja Kuzmanovic,
Profession: Artist/Researcher
Function: FOAM Director
Institutional Affiliation: FOAM/Starlab
E-mail: [email protected]
URL: www.deepfoam.org

Short Biography

Maja has completed a Master of Arts in Interactive Multimedia (with credits), and her
specialization is in Interactive Film and Storytelling. As Artist in Residence at CWI (Center
for Mathematics and Computer Science) in Amsterdam, the Netherlands and GMD (National
Center for Information Technology) in Sankt Augustin, Germany, she worked on projects
concerning innovations in the Internet- as well as the VR technologies. Among other, she was
a part of the discussions about SMIL (Synchronized Multimedia Integration Language), d-
HTML, XML and WAP and has researched the possibilities of storytelling in immersive
Virtual Reality (CAVE). Kuzmanovic is involved in the development of the Design
Technology course at the Utrecht School of the Arts, where she teaches Wearable Computing
and Hybrid Architecture. As the director of the newly formed Starlab's spin-off Foundation of
Affordable Mysticism (FOAM), she works with various art and technology collectives and is
exploring novel modes and resources of cultural expression.

More information: www.cwi.nl/~maja