IEEE Robotics & Automation Magazine - June 2021 - 97

these issues by proposing a set of system performance evaluation
metrics that demonstrate that the proposed methodology
leads to a well-functioning system. In addition, we define an
experimental protocol to compute these metrics. Finally, we
evaluate the ability of our methodology to function as a predictor
of integration time.
Experimental Protocol
The goal of the protocol is to assess the robustness, reliability,
and operational speed of the bimanual manipulation system
for the given packing task. Four tools were identified as representative
examples of a maintenance technician's toolset: a
torch, cutter, brush, and screwdriver. The selected toolbox is a
55 × 25 × 28 cm Stanley Open Tote bag.
To facilitate the evaluation of the system, the task is divided
into two phases: grasping and placement. The grasping
phase consists of grasp planning and the execution of grasp
strategies. The grasping phase ends when the tool loses contact
with the supporting surface, i.e., when the end effector
supports the full weight of the tool. The placement phase
comprises the planning and execution of the arm motion,
from the moment when the hand grasps and lifts the tool up
to the release of the tool inside the toolbox. A single pick-andplace
attempt is considered successful if the tool is in the toolbox
after the placement phase.
We describe the experimental procedure for a single execution
of the given task. As a first step, all four tools of the object
set are placed on the bench within the workspace of each arm.
Tools are placed randomly, but it is ensured that there is more
than a 3-cm gap between them to prevent the creation of clutter.
The robot arms move to an initial predefined position and
autonomously pick and place the tools one by one in an arbitrary
order until there is no pick option available. The system
considers that there are no pick options if 1) there are no more
tools on the bench, 2) the remaining tools are not detected by
the system, or 3) the system is unable to find a feasible way of
grasping any of the remaining tools. No external intervention is
allowed during the execution; therefore, objects dropped out of
the robots' workspace are not reintroduced.
Metrics for System Evaluation
We benchmark the system performance over 10 consecutive
executions of the task. H denotes the set of robot hands used
in the system, and T identifies the set of tools for the experiment.
For this experiment, {}
and {}.
H= qb hand,armar-6hand
T= torch, cutter, brush, screwdriver
As a next step, the task performance metrics is defined. This
t T! If i denotes the trial number, then
.
● n ht, ! 01
s
i
● n ht, $ represents the number of pick and place attempts
{, } indicates if tool t was left on the bench at the
fully pick and place tool t
a
i
● n ht, ! 01
u
i
end of a trial.
The values from each trial can be used to define successbased
metrics, which reflect how successful a hand is at
{, } indicates whether hand h managed to successassesses
how successful a specific hand h H! is at grasping
tool
grasping a specific tool. In particular, the grasping precision is
defined as
P ,,ht and the attempt and success rates are defined
as A ,ht and R ,,ht respectively. The precision score corresponds
to the robustness of a hand's grasping and placement strategies.
A high precision score means that the system is able to
ensure that, if a grasp affordance is detected, the hand-arm
combination can successfully exploit it. The attempt and success
rates represent, respectively, the effectiveness of a system
at detecting a grasp affordance and the effectiveness of the
entire pick-and-place pipeline. As such, for N trials, these
metrics can be formulated as follows:
N
P =ht,
A
R
/
i =1
N
ht, = /
i =
N
ht, = /
i =
n
n
s
i
a
i
ht
ht
,
,
,
1 ,,
,
nn
n
a
i
a
i
a
i
1 ,,
,
nn
n
s
i
ht
ht +
u
i
.
ht
Finally, the overall system performance is evaluated based
on the mean picks per hour (MPPH) and the task completion
score, S. MPPH is a well-known metric in logistics for measuring
both human and machine efficiency and throughput.
The task completion score corresponds to the average number
of tools that were successfully picked and placed per trial, i.e.,
S = 1
N
/
iN ht1
[, ],
!! !,H T
System Evaluation Results
This section presents and analyses the benchmark results for
our system. The success of a complex manipulation task
depends both on the strategies used (grasp planning, robot
control, and so on) and on the effective integration of every
component involved. However, a poor integration is likely to
have a visible negative effect on the overall system performance.
Therefore, in the following discussion, we identify
what affects the performance of our system, and we demonstrate
that our integration methodology does not cause any
system failures but, on the contrary, facilitates the development
of a well-functioning system.
According to the overall system performance metrics
(Table 6), our proposed integration methodology leads to a
system that is able to complete more than 75% of the given
packing tasks on average. We should note that the evaluated
system is still in the research stage and is not optimized for
speed of operation as would be required in production. As a
consequence, the MPPH is quite low, even in the cases where
the system manages to fully complete the task, i.e., S = 8.
To facilitate system introspection and identify potential
integration issues, we monitored system failures during task
execution. More specifically, we identified the following failure
types.
●
Visual detection failure: False positive and negative objects
as detected by the Mask R-CNN model.
JUNE 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
97
n
s
i
ht,
.
(6)
ht
ht +
u
i
,
ht
(5)
(4)
(3)
IEEE Robotics & Automation Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2021
