IEEE Technology and Society Magazine - March 2022 - 35

Figure 3. Hybrid 2-D/3-D model of
wharerau interior.
The Oculus head-mounted display system allows
the user to physically and simultaneously virtually
walk within a 3-m circle. For navigation beyond this
circle, as it is necessary in our ~14 m in diameter
wharerau, we implemented a simple teleportation
mechanism (see Figure 4, left). Pointing at the virtual
floor and pressing the controller button will display a
" cardboard figure " to indicate where one would be
teleported to, while twisting the controller changes
the figure's orientation indicating the user's target
orientation. With this interface, any viewing angle
can be achieved within the environment, including
closeup views of the artwork.
Figure 4. Left: Inside view including
teleportation mechanism. Right: user
wearing head-mounted display while
exploring the virtual wharerau.
and MeshLabiii were used to 3-D reconstruct this part
of the building and the resulting 3-D mesh component
is combined with the rest of the 3-D model
(see Figure 3).
Finally, in a third iterative step, the entire interior
architecture of the wharerau is photogrammetrically
reconstructed from thousands of photographs,
including 3-D structures (e.g., additional four t ı¯puna
figures around a central column). The resulting
model had to be simplified-reduction of the number
of triangles to be rendered-so that a real-time,
immersive experience is achieved. The photogrammetric
reconstruction and model reduction were
computed using RealityCapture,iv a proprietary payper-pixels
software, and the immersive experience
was implemented using Unreal Engine (v4.26.2).
The virtual wharerau can be experienced on any
computer which has a dedicated GPU with at least
4-GB video RAM (VRAM) and a modern CPU with at
least 8 GB RAM. While the software is supporting a
wide range of VR peripheries, we are using an Oculus
Rift CV1 or S for our system (Figure 4, right).
iiihttps://github.com/colmap/colmap
ivhttps://www.capturingreality.com/
We informally tested the experience for feasibility,
usability, and felt presence in the virtual wharerau.
Different user groups were exposed to the system,
Ma¯ori and Pa¯keha¯: experienced and first-time users
of VR, people who knew the real wharerau and had
been there before, and people who had never been to
this marae or to any marae at all. The Pa¯keha¯ partners
have been invited to the marae many times and are
now considered as " part of the wha¯nau (family). " This
allowed the system to be presented at a very important
event, the Waitangi Day celebration, at the marae
with over 500 visitors attending.
It was surprising and satisfying to see that almost
all people felt a self-reported sense of being there at
the wharerau and that even novice users had little to
no difficulty in exploring the virtual building. A few
people reported light dizziness during or after use,
but many felt disoriented after (abruptly) doffing the
head-mounted display. Also, the teleportation user
interface normally needed instruction and a couple
of minutes to get used to. In addition to reported
excitement about the system and its (potential and
current) capabilities, we also observed behavior
very similar to real-world behavior in a wharerau.
Overall, the desired experience and feasibility
could be validated by our informal studies, as well
as at presentations and gatherings outside the laboratory
since those laboratory evaluations. After
achieving the enabling sense of presence in the
virtual wharerau, we added virtual storytelling to
the experience. To maintain the sense of presence
[13], we opted for the integration of 3-D, volumetric
videos in the form of voxels in the building. These
voxelvideos [10] allow for a coarse " holographic "
in-space visualization of a storyteller, including
March 2022
35
https://github.com/colmap/colmap https://www.capturingreality.com/
IEEE Technology and Society Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2022
