IEEE Robotics & Automation Magazine - September 2015 - 48

have developed a number of example protocols to both provide more concrete samples of the types of task definitions
that can be put forward as well as specific and useful benchmarks for actually quantifying performance. We have defined
five protocols to date:
● pitcher-mug protocol
● gripper-assessment protocol
● table-setting protocol
● block pick-and-place protocol
● peg-insertion learning-assessment protocol.
From each protocol, a benchmark of reported performance is derived with the same name. We have implemented
each of the protocols experimentally and reported
the benchmark perforA URDF file that spawns
mance of our implementations for each. All these
the scenario for Gazebo
protocols and benchmarks and the results dissimulation environment
cussed in this section can
be found at [64]. We have
is given.
also implemented the
box-and-blocks test for
maintaining a baseline performance of this test for robotic
manipulation.
YCB Pitcher-Mug Protocol and Benchmark
One of the popular tasks among robotics researchers is pouring a liquid from a container. This task is interesting as it necessitates semantic interpretation and smooth and precise
manipulation of the target object. A protocol is designed for

Figure 12. The HERB robot implementing the pitcher-mug
benchmark.

(a)

(b)

Figure 13. The grippers compared with gripper assessment
benchmark: (a) Model T and (b) Model T42.

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

executing this manipulation task. The protocol uses the pitcher and the mug of YCB object and model set and provides
scenarios by specifying ten initial configurations of the pitcher and the mug. By standardizing the objects and providing
detailed initial state information, it aims at maintaining a
common basis of comparison between different research
groups. The benchmark derived from this protocol uses a
scoring scheme that penalizes the amount of liquid that remains in the pitcher or spilled on the table. This benchmark
was applied using the HERB robot platform [61], which can
be seen in Figure 12. The reported results show that the task is
successfully executed for eight of ten pitcher-mug configurations. For the two failed cases, the robot is able to grasp the
pitcher but cannot generate a suitable path for pouring the
liquid. This shows the importance of planning the manipulation task as a whole rather than in segments.
YCB Gripper-Assessment Protocol and Benchmark
The abilities of a robot's gripper affect its manipulation performance significantly. In the literature and in the commercial
market, various gripper designs are available, each of which
has different manipulation capabilities. The protocol defines a
test procedure for assessing the performance of grippers for
grasping objects of various shapes and sizes. This protocol utilizes objects from the shape and tool categories of the YCB
object and model set. Using this protocol, a benchmark is defined based on a scoring table. We applied this benchmark to
two grippers designed in Yale GRAB Lab, the Model T and
Model T42 [65], which are shown in Figure 13. The results
show that the Model T can provide successful grasp for only a
limited range of object sizes. This gripper is not suitable for
grasping small and flat objects. However, the ability to interlace its fingers increases the contact surface with the object
and brings an advantage, especially for grasping concave and
articulated objects. The Model T42 is able to provide stable
power grasps for large objects and precision grasps for small
objects. This model is also successful in grasping flat objects
thanks to its nail-like fingertips. However, not being able to
interlace its fingers brings a disadvantage while grasping articulated objects. Using the same benchmark for evaluating different gripper designs not only provided a basis of
comparison but also gave many clues about how to improve
the designs.
YCB Protocol and Benchmark for Table Setting
Pick-and-place is an essential ability for service robots. The
benchmark assesses this ability by the daily task of table setting. The protocol uses the mug, fork, knife, spoon, bowl, and
plate of the YCB object and model set. These objects are
placed to predefined initial locations, and the robot is expected to replace them to specific final configurations. The benchmark scores the performance of the robot by the accuracy of
the final object poses. This benchmark can also be applied in
a simulation environment, since the models of the objects are
provided by the YCB object and model set. A URDF file that
spawns the scenario for Gazebo simulation environment is

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015