IEEE Technology and Society Magazine - March 2022 - 37

people immersed in the virtual wharerau experiencing
not only a sense of presence (being there in the
environment), but also a sense of co-presence (being
together with the storyteller). This means that, even
though the virtual storyteller is not really " there, " the
viewers still treat him/her as though he/she is. That is,
they avoid " bumping into " or " walking through " the
virtual storyteller and will also not leave the virtual
room before the storyteller has finished speaking.
In addition, we noticed with some people the reluctance
to interrupt the storyteller, taking off their shoes
before entering the virtual wharerau, or confirmatory
spoken words during storytelling " Kia ora. "
The main shortcomings of our approach are the
relatively low resolution of the voxels (4-8 mm)
and the lack of true interactivity with the storyteller,
since it is a recorded story. The low resolution did
not prevent viewers from developing a sense of
co-presence: the quality was sufficient for this feeling
to develop. However, a number of respondents
were uncomfortable with aspects of the " ghost-like "
voxelized representations which enabled the participants
to see through the storyteller's head (a sacred
body part). The low resolution also made it difficult
to judge facial expressions of the storyteller, so a
higher quality is still desirable. The lack of interactivity
with the storyteller can be addressed by either
algorithmically animating the storyteller's postural
and gestural
behavior
as
video-game
behavior
engines would provide, or by providing a real-time
system with a storyteller recorded/captured at the
same time as the receiver is experiencing the story.
The latter option is supported by our voxel tele-copresence
approach and system.
Three-dimensional tele-co-presence
Our approach to tele-co-presence with our voxel-based
system stems from two affordances:
* to make the storytelling experience interactive
in real-time for both the storyteller and the story
receiver, and more importantly
* to connect dispersed Ma¯ori to their marae, if they
are unable or hesitant to do so physically.
We extend the voxelvideo recording and playback
aspect of our system toward real-time transmission of
voxel streams from one geographic location to another.
With such an approach to interactive storytelling, it
does not matter whether the storytelling participants
are in the same region or on opposite sides of the earth.
March 2022
Two or more participating people are wearing
immersive, tracked head-mounted displays
and are simultaneously captured by a minimum
of three RGB-D cameras at each location. Each
party's computer processes the camera data to
form a voxel stream which is then transmitted to
the other party. All parties also have 3-D models
of the virtual place to meet in, here our wharerau
model, including its artistic interior. The real-time
voxelvideos are placed within the virtual model
together with optional prerecorded voxelvideo
stories and other 3-D artifacts. All systems are synchronized
so that each participant has the impression
of being in a shared environment with the
others (see Figures 6 and 7).
The desired outcome is the feeling of tele-copresence
for all participants. The " tele-presence "
part means the feeling of " being there " in the
virtual marae, that is, of being at Te Rau Aroha
(there in Bluff). Additionally, the " co-presence "
part means the feeling of being together [17], sideby-side
with the other participants and with the
prerecorded storyteller as well. While tele-presence
alone means that one is present in a remote
place, tele-co-presence is the feeling of presence
in that space with another person or people. Our
system also allows for variations on tele-presence:
instead of meeting in a virtual space (in our case,
a beforehand 3-D reconstruction), participants can
also meet in each other's physical environment, as
it can be voxelized and transmitted in addition to
the user's body.
To date, we have informally tested and evaluated
our tele-co-presence approach with studies
within two adjacent laboratory rooms, as well as
at public expositions and at our partner marae,
but also between two cities in Aotearoa, and even
as far as Dunedin-Aotearoa/NZ-London-U.K. as
a proof of concept. In the latter test, one has to
cope with the latency of about a half a second
occurring because of the speed (of light) of the
transmission. The main restriction with our immersive
tele-co-presence approach lies in the use of
head-mounted displays which obscure the participants'
faces and the limitations of fine gestural
expressions imposed by the coarse voxel resolution.
However, we could observe that larger gestures
and general posture expressions work very
well (like pointing, body swaying, greetings, and
hugging).
37
IEEE Technology and Society Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2022
