IEEE Robotics & Automation Magazine - September 2021 - 68

toward U-HRI is the detection and tracking of one or multiple
divers [3]-[13]. Given the relative localization, different
protocols for interaction can be studied and trained in a computer
simulation [14].
Relative motions between divers and robots can already be
used for a basic nonverbal form of communication [15], but
more capable forms of communication-in terms of expressiveness
and reliability-are needed to enable real U-HRI for
collaborative missions. Work in that direction is described in
[16], where artificial fiducial markers are used that are then
interpreted by the robot using grammatical rules. While cards
with artificial markers ease the challenges of underwater
vision, there are disadvantages like the number of cards that
the diver must carry and the effort required to handle them.
Gestures are a more natural basis for underwater communication
because 1) they are already extensively used by divers
and 2) they operate despite the limitations of water as a medium,
e.g., it is impossible to use voice recognition. Early
research on the use of gestures for U-HRI is described in [17],
where waving gestures are recognized by differential imaging
with a spectral registration method in the form of the
improved Fourier Mellin invariant. Based on that, trajectories
of hand motions are recognized with a finite state machine
(FSM). The experiments in [17] are done in a pool.
An imaging sonar, also known as an acoustic camera, is
used in [18] for gesture recognition. Preprocessing stages with
cascade classifiers and shape processing are combined with
three different classification approaches, namely, a convex hull
method, support vector machines, and the fusion of both.
Experiments are conducted in a pool as well as during field
trials with divers in the context of the EU project Cognitive
Autonomous Diving Buddy (CADDY) (Figure 1). The selection
of device parameters within a mission is a known challenge
for this type of sensor, which is also reported in [18].
The main type of sensor for U-HRI in CADDY is, therefore,
a (stereo)camera. To process the visual data, a modification
of nearest class mean forests (NCMF) in the form of a
multidescriptor extension (MD-NCMF) is introduced. MDNCMF
is used for both diver detection and tracking [8] and
the classification of diver gestures [19]. As the name suggests,
MD-NCMF is designed to exploit different types of descriptors
to achieve high robustness under the challenging conditions
of underwater visibility. To this end, MD-NCMF builds
on NCMF, which partitions the sample space by comparing
the distances between class means instead of comparing values
at each feature dimension, as in more traditional random
forests approaches. Therefore, MD-NCMF can treat each feature-object
pair as a new class, e.g., speeded up robust features
(SURF) object1, scale-invariant feature transform (SIFT)
object2, SURF object2, SIFT background, and so on, and
MD-NCMF can examine which one provides the best partition
of the sample set.
Based on MD-NCMF gesture recognition [19], a machine
USBL
Underwater
Tablet
Stereo
Camera
Sonar
interpreter [20] with a phrase parser, syntax checker, and
command dispatcher linked to the mission control allows
the use of a very expressive language for U-HRI [21]. This
Caddian language is based on a context-free grammar that
allows the diver to specify missions with a sequence of
tasks. The syntax checker is implemented as an FSM that
gives constant feedback to the diver and allows for in situ
corrections. The gesture recognition front end and the
machine interpreter back end are reported in field tests to
be not only robust but also useful in complex missions with
professional divers [22], [23].
A full language for U-HRI is also presented in [24]. It is
PlaDyPos
Surface Vehicle
Diver
Figure 1. The CADDY system for assistance in diver missions.
(Inset) The Buddy AUV is equipped with a Blueprint Subsea
X150 ultra-short baseline (USBL), an underwater tablet, a
BumbleBeeXB3 stereo camera, and an ARIS 3000 imaging sonar
for diver tracking, monitoring, and communication. In the top
portion of the image, a diver gestures a command, and the lower
portion of the image shows an aerial view of the system with a
PlaDyPos surface vehicle for global positioning.
68 * IEEE ROBOTICS & AUTOMATION MAGAZINE * SEPTEMBER 2021
Buddy
Underwater Vehicle
syntactically a bit simpler than Caddian, as the FSM in its
interpreter is restricted to only one possible transition from
state to state, i.e., gesture to gesture, to avoid ambiguities. The
gesture recognition front end in [24] is based on DL models.
More precisely, the single shot detector (SSD) [25] and faster
region-based convolutional neural networks (faster R-CNNs)
[26] are investigated, which achieve more than 90% accuracy
when being trained with a data set of 50,000 points.
It is assumed in [24] that the diver wears no gloves; this
enables the use of skin detection and image contour estimation.
In practice, professional divers tend to always wear
gloves, both for protection and to avoid heat loss. For the
MD-NCMF gesture recognition [19] mentioned previously,
regular diving gloves are augmented with colored stripes to
provide some detectable contrast. The first results toward a
classification under a wide range of conditions, including divers
with and without gloves, are presented in [27]. Building
upon a DL-based approach dubbed SCUBANet to recognize
diver body parts [28], MobileNetV2 [29] is trained to
IEEE Robotics & Automation Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2021
