IEEE Robotics & Automation Magazine - September 2021 - 19

which corresponds to the path described by the door handle.
While opening the door, the IM is defined such that the user
essentially needs to create control signals in only a single DoF,
namely, the direction toward the door handle. As a result, the
complete 6D Cartesian control of the end effector is automatically
handled by the shared-control algorithm.
To enlarge the kinematic reachability of the arm, especially
in tasks that require a large range of motion, EDAN is extended
with a whole-body-control approach [22], which coordinates
the motion of the wheelchair and manipulator, either in
shared-control tasks or even in direct control if desired by the
user. Especially in an assistive system, such as EDAN, wholebody
control allows the performance of tasks that require a
coordinated motion of the mobile base and the manipulator,
e.g., when opening a door and passing through.
In EDAN, a variety of shared-control skills are implemented
to support the execution of everyday tasks. In our framework,
shared-control skills are object-centered and stored in
an object database (ODB) in a human-readable format. In
addition to skill definitions, the ODB also provides information
such as name, class affiliation, geometry, or grasp and
tool frames for each object. Objects stored in the ODB can be
localized using the RGB-D camera(s) and a perception pipeline
based on a bounding box detector followed by a poseestimator
algorithm, as described in [23]. For objects with a
support plane, such as doors or drawer handles, the plane
equation is estimated from depth data and intersected with
the object bounding box for a more refined pose.
Once an object has been detected within the reachable
workspace, it is stored in the robot's world representation,
and shared-control skills associated with the object become
available. The user can choose to activate a skill directly
from EDAN's tablet computer or have it automatically activated
as soon as the end effector is moved into close vicinity
of the object of interest. Accordingly, the user can abort
the currently active task by moving away from the object of
interest. Information about available tasks and possible
activation are visualized on the tablet computer.
Supervised Autonomy
Direct and shared control are suitable modes for assistive
systems, such as EDAN, which follow a human-in-the-loop
approach. In service robots, like Justin, a different strategy is
required since the robot is supposed to execute tasks without
continuous human interaction. As a result, the system needs
to operate on a higher level of autonomy, and, simultaneously,
the user interface must work on a higher level of abstraction.
This leads to the control mode of supervised autonomy
as it is employed in the Justin system.
Based on the findings obtained during the Multipurpose
End-to-End Robotics Operation Network (METERON)
Justin experiments [20], the interface, initially designed for
astronaut-robot teams, is repurposed for elderly care scenarios.
The user interface relies on highly abstract commands
generated by the robot autonomy layer. This system
uses an object-centered knowledge representation to plan
the execution of a task.
The knowledge about objects in the environment is stored
in our common knowledge base ODB. This knowledge base
contains object-specific action templates [24] that provide the
robot with symbolic and geometric descriptions of the respective
manipulation tasks. For application in elderly care, we
have added numerous objects and action templates to the
ODB to allow the robot to interact with its new environment.
A hybrid planning framework utilizes this information to
solve the given task symbolically and find a suitable geometric
solution for the intended course of action [Figure 5(c)]. The
planning is carried out according to the actual symbolic and
geometric states of the environment and uses robot-specific
planning modules, such as motion planning and controller
parameterization [25]. If the robot does not find a suitable
geometric solution, the action is re-evaluated by means of geometric
and symbolic backtracking to find alternative solutions.
In addition to symbolic and geometric planning, another
key aspect to achieving autonomous operation is selflocalization
and navigation (Figure 6). In Justin, a 3D map
of the indoor scene is created for localization purposes.
(a)
(b)
Figure 6. Localization based on the 3D model of the environment. (a) A photo of the actual scene. (b) A rendering of the robot
localized in the kitchen area. The colored dots below the robot and in the inset bird's eye view depict the hypothesis of the localization.
SEPTEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
19

IEEE Robotics & Automation Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2021