Tech Briefs Magazine - November 2021 - 45

Why 3D Artificial Intelligence with AR and VR
is More Important Today
T
he race continues among the
world's largest tech leaders and
companies to see which one will
prevail and power the next generation
of tools, technologies, and re -
sources for manufacturing, healthcare,
construction, and many other vertical
market software applications.
These companies have been working
tirelessly to create changes that will
make a significant impact on our world.
This all starts with the technological
advances that have been made in recent
years through software powered by
Artificial Intelligence (AI) and immersive
mixed reality technologies such as
Augmented Reality (AR) and Virtual
Reality (VR).
There are specific differences among
all these technologies but today, they are
working together in advanced threedimensional
(3D) applications and environments,
all for the benefit of these
companies and their customers.
Immersive Mixed Reality Uses
With virtual reality, a user wears a
headset that fully delves into a new
world or environment - some that even
mimic the real world. The user is given
both a visual and audible experience
that is meant to replicate a real-world
setting in a manufacturing environment.
Augmented reality is similar in concept
but it also displays digital content in
the real world. Imagine an automotive
manufacturer where an engineer holds
up an iPad in front of a car being
designed to see virtual specs of a vehicle's
design layout or various engine
spec options.
The challenge is that these technologies
require heavy doses of data, the ability
to process vast amounts of data at
incredible speeds, and the ability to
scale projects in a computer environment
that doesn't often fit traditional
office environments.
Immersive mixed reality requires a
precise and persistent fusion of the real
and virtual worlds. This means rendering
complex models and scenes in phoTech
Briefs, November 2021
Cov
www.techbriefs.com
ToC
torealistic detail, rendered at the correct
physical location (with respect to both
the real and virtual worlds) with the correct
scale and accurate pose. Think of
the accuracy and precise nature re -
quired in leveraging AR/VR to design,
build, or repair components of an aircraft
engine or an advanced surgical
device used in medical applications.
This is achieved today by using discrete
GPUs from one or more servers
and delivering the rendered frames
wirelessly or remotely to head mounted
displays (HMDs) such as the Microsoft
HoloLens and the Oculus Quest.
The Need for 3D and AI in Immersive
Mixed Reality
One of the key requirements for
mixed reality applications is to precisely
overlay on an object its model or digital
twin. This helps in providing work in -
structions for assembly and training and
to catch any errors or defects in manufacturing.
The user can also track the
object(s) and adjust the rendering as the
work progresses.
Most on-device object
tracking systems
use 2D image and/or marker-based
tracking. This severely limits overlay
accuracy in 3D because 2D tracking cannot
estimate depth with high accuracy
and consequently, the scale and the
pose. This means even though users can
get what looks like a good match when
looking from one angle and/or position,
the overlay loses alignment as the
user moves around in six degrees of freedom
(DOF).
The object detection, identification,
and its scale and orientation estimation
- called object registration - is
achieved, in most cases, computationally
or using simple computer vision
methods with standard training li -
braries; for example, Google Media Pipe
45
http://www.techbriefs.com http://info.hotims.com/79419-902
Tech Briefs Magazine - November 2021

Table of Contents for the Digital Edition of Tech Briefs Magazine - November 2021
