IEEE Robotics & Automation Magazine - September 2021 - 20

During operation, recently obtained sensor readings are
compared to this map. This is done by sampling multiple
random pose hypotheses based on the last position of the
robot and the commanded movement. For each hypothesis,
the expected sensor output (here, depth images) is generated
and compared to the actual measurements. Since Justin
is equipped with four RGB-D sensors, redundant sensor
information is available. This feature can be exploited to
resolve ambiguous situations and increase the robustness of
the indoor localization. In the end, the most probable pose
hypothesis is selected as the current pose.
With the obtained pose, the navigation module can plan
a collision-free path to a selected destination. This target
can be either directly selected by a human or part of the
autonomy of the robot. Here, a standard A-star search
algorithm is applied to plan the shortest path to the target.
The algorithm works on a graph that connects all safely
traversable positions and takes the configuration of the
robot into account because, for certain areas, the arms may
have to be folded.
Based on the action templates and navigation skills, the
Justin system can be commanded with a high degree of
abstraction, where available actions depend on the current
state of the environment and objects relevant to the task,
which are stored in the world representation. We continued
to use the autonomy system from the METERON experiments
unchanged, as the object-centered knowledge representation
has proven to be useful and easy to translate into
the user interface.
Autonomous Manipulation
Supervised autonomy makes it possible to adjust the level
of autonomy of the robotic system as well as the complexity
of the interface to the preferences and capabilities of the
user. However, for some tasks, like searching for a specific
object, a higher level of autonomy, e.g., in combination
with voice commands, may be desirable. To enable a robot
to perform a task fully autonomously, 3D visual perception,
fast whole-body motion planning, and their close
interaction are crucial. This is particularly true in household
environments with challenging and ever-changing
geometrical constraints.
To allow for this, a precise 3D environment model is
generated in real time (30-Hz frame rate) using GPU-based
probabilistic fusion algorithms [26], [27] that integrate the
stream of depth images from the head-mounted Kinect
camera into a dense voxel model, which reaches a precision
of 2 mm in a typical manipulation distance of up to 2 m.
These 3D models are the basis for an optimizationbased
motion planner that determines a collision-free
motion for all 19 upper-body DoF and the mobile platform
in less than 1 s [28]. To speed up the solver convergence
and avoid local minima, we developed an informed multistart
approach that draws guesses for the initial trajectories
using a generative neural network (GAN). The GAN is
conditioned on the current 3D model as well as the start
20 * IEEE ROBOTICS & AUTOMATION MAGAZINE * SEPTEMBER 2021
and target configurations, and it is trained in batch mode
from numerous example planning problems.
The high-resolution 3D model is also used for the pose
estimation of objects to be grasped and manipulated. Our
algorithm predicts a number of point correspondences to a
given 3D model of the target object. From a moderate
number of correct predictions, the object pose can be estimated
using classical robust techniques [such as random
sample consensus (RANSAC)] even if contaminated with a
high number of outliers. Multiple pose hypotheses are generated
from groups of predicted point correspondences and
evaluated for their fit to the scene model, thus optimizing
the three rotational and three translational DoF.
Because a large number of point correspondences are
usually predicted for each scene, not all of them being correct,
a prediction method is required that delivers a meaningful
confidence measure along with each prediction.
With this consideration in mind, we decided to use
extremely randomized trees for the correspondence prediction,
a method previously used in a similar problem setting
[29]. Only high-confidence correspondences are then
regarded in the generation of pose hypotheses.
To be able to autonomously find a target object from large
distances, we implemented a combined exploration and
detection strategy. For the object detection, a RetinaNet [30]
detector was trained on various appearances of the target
objects in RGB images, viewed from distances of about 1-3 m
and under various lighting conditions. Combined with our
real-time 3D environment modeling and fast motion-planning
algorithms, the robot is able to move around in a room
while streaming RGB images at 1 Hz to the detector in search
of the target object. For the object detected in an RGB image,
a synchronous depth image is used to transform the image
coordinates of the object into 3D space, i.e., the target location
for the robot for fetching the object.
Telepresence
In addition to the control modes designed for users of the
robots, the ecosystem provides the opportunity of control
via telepresence by an expert, utilizing the sensory channels
provided by HUG (see the " HUG " section). A high level of
immersion or transparency, respectively, enables the operator
to perceive the remote environment with his/her own
senses in such a way that it feels like being on the remote
site. The senses generally considered in telepresence systems
provide visual, audio, and haptic (kinesthetic and tactile)
information of the remote environment.
In the ecosystem, DLR HUG is used as the main telepresence
device, providing the operator with the required
manipulation and sensing capabilities. Using HUG, the
operator can control all DoF of the avatar system and
simultaneously feel the forces exerted on the environment,
which are visually perceived through a pair of stereo cameras
mounted on the avatar.
For safe and high-performance teleoperation, the quality
of haptic and visual feedback is known to be more relevant
IEEE Robotics & Automation Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2021
