IEEE Robotics & Automation Magazine - June 2021 - 62

This is motivated by the high costs of in-store logistics, up to
60% of the total operational store costs [4].
The most time-consuming task is, certainly, shelf replenishment,
and 50% of such time is devoted to finding the correct
slot on the shelf. Only a few scientific contributions exist
on this specific automation issue [5]-[7], mainly re lated to the
Future Convenience Store robotic challenge launched by the
World Robotic Summit [8]. Most of the solutions are based
on the use of vacuum grippers; thus, they are limited to those
cases where pick-and-place grasp poses are the same. On the
other hand, the shelf replenishment task in a real environment
may require sophisticated manipulation skills.
The Robotics Enabling Fully Integrated Logistics Lines for
Supermarkets (REFILLS) project [9] proposes a complex software
architecture to execute complete shelf replenishment
with a speed comparable to that of a human. This includes a
task-planning framework based on KnowRob [10].
The present article focuses its contribution on the lower
hierarchical level. This comprises an object-detection module
based on state-of-the-art algorithms; two innovative reactive
controllers, one for visual servoing and one for grasp control;
and a geometric-motion planner tightly connected with the
reactive control modules.
Object detection and localization are the first steps of any
pick-and-place task. The most recent and advanced localization
methods, such as DenseFusion [11] and PoseCNN [12], are still
not accurate enough for the in-hand maneuvers described in
this article. Therefore, we adopted a slightly different approach
based on two steps. First, we detect the object by resorting to a
well-known convolutional neural network (CNN) [13] and
simply use the bounding box information together with the
depth map to infer a rough 3D object position. Then, we use a
visual-servoing controller to grasp the object, the reference
image of which has been previously stored in the object database
together with the grasp positions.
Planning a grasp location suitable for moving the object
from the pick pose to shelf facing, with the correct pose, is a
challenge for two reasons. First, the grasp pose should be such
that the final placing pose is reachable without any collision in a
cluttered environment and in narrow spaces like those of a supermarket
shelf. Second, the grasp location should allow in-hand
manipulation actions that may be necessary to perform the task.
For instance, placing a bottle vertically can be accomplished even
if the bottle is knocked over on the table; it is sufficient to grasp
it far from its center of gravity (CoG) and then pivot it in hand.
However, the grasping force has to be correctly established.
It is essential for performing such in-hand manipulations,
and it should ensure a stable grasp during the
transportation while avoiding any deformation or damage.
We believe that tactile perception is the enabling technology
for achieving these control objectives. Contact information useful
for manipulation ranges from simple contact detection to
spatially distributed pressure sensing and contact-force and
moment measurements.
Many options are currently available, from piezo-resistive
to optical sensors and from capacitive to magnetic devices;
the reader is referred to the comprehensive surveys on this
kind of technology [14]. The tactile sensor used in this work
is able to measure the contact wrench, including torsional
moment, and it was developed in our laboratory [15].
This article presents the latest results of the REFILLS project,
which includes the complete reactive control layer endowed
with the grasp controller based on the results in [16]-[18] and a
visual-servoing controller that allows the safe grasping of objects
in an uncertain and dynamic context. Eventually, a motionplanning
algorithm, based on the MoveIt! planning framework
[19], is proposed with a twofold innovation. First, the manipulator
motion is planned by taking into account the manipulation
abilities provided by the low-level grasp controller. Second, the
motion planner computes, for each motion segment of the pickand-place
task, the most appropriate grasp control modality,
chosen between slipping avoidance and controlled sliding.
The lab-scale shelf replenishment setup includes a pick desk
where the items to restock are randomly arranged, emulating a
trolley prepared in the back room by a human operator that
autonomously navigates in the store toward the shelf to
replenish. The shelf constitutes three layers with separators
identifying the slots where the products need to be arranged
(see Figure 1). The high-level robot control system, directly
interfaced with the store management system (SMS), knows
which products are present on the pick desk. The SMS asks
the robot to pick a specific item to place on an assigned slot.
A similar task is the objective of the Future Convenience
Figure 1. The shelf replenishment setup: a KUKA LBR iiwa robot,
a Schunk WSG50 gripper, SUNtouch tactile fingers, an Intel D435i
camera, a pick desk, and the shelves to be refilled with three
facings each.
62 * IEEE ROBOTICS & AUTOMATION MAGAZINE * JUNE 2021
Store challenge. In contrast to the operating conditions of the
challenge, where the clearance between two shelf layers is
250 mm, in our setup, the clearance is only 150 mm, and we are
not allowed to use extractable shelf layers to remove collision
points with other layers. This makes the task even more difficult
since placing from the top is possible only on the first layer.
Other relevant differences include the presence of 5-cm-high
separators between the facings of the shelves and the fact
that we are not allowed to place augmented reality tags to
localize the objects, so we have to rely on the object texture only.
IEEE Robotics & Automation Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2021
