IEEE Robotics & Automation Magazine - September 2021 - 69

recognize 25 image classes using finger count and palm direction,
although the authors also state that a significant portion
of these classes is unused in most gestures [27].
U-HRI With Gesture-Based Communication
Our gesture-based communication for U-HRI consists of a
gesture recognition front end and an interpreter back end. In
this article, different options for the front end are investigated,
as described in the " Gesture Detection and Classification " section.
A short overview of the actual language and the interpreter
back end is then given in the " The Caddian Language and
Its Interpretation " section. An example from a field trial in the
" Example Use Case and Challenges " section illustrates the use
of the complete system and the challenges that occur in practice
and motivates the investigation of different DL methods.
ML in general and DL in particular typically require high
amounts of data for training and evaluation. A physically realistic
way to use the range information for adding underwater
haze is hence introduced in the " Physically Realistic Underwater
Image Degradation " section. This is used to add artificial
degradations to existing real-world images from field
trials to produce additional data, which are useful for covering
the high amount of variability in the underwater domain
without the need for many costly field campaigns.
Gesture Detection and Classification
MD-NCMF as a Classical ML Approach
MD-NCMF is a multidescriptor extension of NCMF that is
used for both diver detection/tracking and the classification of
diver gestures [8], [19] (Figure 2). This variant of random forests
aggregates multiple descriptors (SIFT, SURF, ORB, HoG,
and so on) that encode different representations of the objects
of interest as we observed that each of these descriptors is
robust to different types of underwater image degradations.
MD-NCMF can be considered to be a classical ML approach,
which forms a comparison basis for the different DL methods
described in the next section.
For the first step of hand detection, both 2D monocular
images and 2.5D stereo disparity are used. The 2.5D disparity
maps are segmented based on distance and density. This provides
reliable hand detection in many cases. However, it fails
for texture-rich interferences close to the stereo camera, e.g.,
due to air bubbles. Therefore, 2D cascade classifiers are used
in a second process running in parallel to filter out the false
positive regions. The resulting region proposals, i.e., object
candidates, serve as the input to the actual classifier. MDNCMF
then filters out further false positives that may still
exist, and it maps the hand regions to the gestures of the
Caddian language described in the following sections.
DL Approaches
State-of-the-art deep models for visual object detection and
classification often follow three meta-architectures: SSD [25],
faster R-CNN [26], and region-based fully convolutional neural
network (R-FCN). SSD models offer fast computation
speeds since they perform object detection and classification
in one single pass of the network. These are, hence, often preferred
for embedded systems. Faster R-CNN has two stages
that are conceptually similar to the described classical ML
approach (see the " MD-NCMF as a Classical ML Approach "
section): a region proposal network generates candidates
for object regions, and
a classifier then verifies
and refines the proposals.
The R-FCN architecture
is a mixture of the previous
two meta-architectures.
It shares features
learned in the initial layers
between the region
proposal and the actual
classifier network.
The DL models are
used with pretrained feature
extractors (Table 1). A
fully connected network
like ResNet [30] can be
considered the most
straightforward approach
since it requires only one
label per image and no
region candidate, which
ultimately satisfies our system's
requirements. The
SSD [25] and faster R-CNN [26] differ mostly in their architectures
among the considered DL methods; the former is tailored
toward fast computation when using the MobileNet
feature extractor [31]. A deformable ConvNet [32] allows
region proposals with nonuniform boundaries by using a flexible
sampling grid on the image. Thus, it is no longer assumed
that the object geometry is fixed, which can be beneficial for
detecting 6-degrees of freedom hands of a free-floating diver.
The Caddian Language and Its Interpretation
The gestures form a language for U-HRI called Caddian [20],
which is derived from the routine communication of divers.
The Caddian syntax defines boundaries to understand complex
commands, i.e., sequences of gestures, which can also be
aggregated to form missions composed of several tasks. Two
gestures to start a command and to end a communication,
denoted as A and
are sequences of individual gestures delimited ( ,)
6 , are used for this purpose. Commands
AA that
represent a single task. A practical example from field trials is
the command " Take a photo at 3 meters altitude. " Missions
consist of aggregated commands that are delimited by (, ).A 6
An example for a mission used in practice is " Take a photo, go
to the boat, and carry the equipment back. "
To handle very frequent tasks or emergencies, there is a
special slang group of gestures. They have higher priority and
a simpler syntax. Examples include a gesture to instruct the
AUV to take a photo at the current location, i.e., without
SEPTEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
69
To this end, MD-NCMF
builds on NCMF, which
partitions the sample space
by comparing the distances
between class means
instead of comparing
values at each feature
dimension, as in more
traditional random forests
approaches.
IEEE Robotics & Automation Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2021
