IEEE Robotics & Automation Magazine - June 2020 - 75

*

(a)

-5
-5 -4 -3 -2-1 0 1 2 3 4 5
Horizontal, tx (mm)

-4

-3

-2

-1

0

1

2

4 MAE = 0.6 mm

3

Predicted Horizontal, tx (mm)

5

Predicted Vertical, tz (mm)
JUNE 2020

Figure 9. The optimal PoseNet performance for the 3D edge. The smoothed predictions (red: 100-sample moving average), region within the smoothed absolute error (pink), and
MAEs are shown. The (a) horizontal, (b) vertical, (c) roll, (d) pitch, and (e) yaw results.

(e)
(d)
(b)

-1
-4
-3
-2
Vertical, tz (mm)
-5
-5

-4

-3

MAE = 0.2 mm
-2

(c)

10 15

-10

-5

0

5

-15
-15 -10 -5 0 5
Roll, θnx (°)

-5

0

5

-10

MAE = 1.6°
10
MAE = 1.4°
10

15

Predicted Roll, θnx (°)

-1

-15
-15 -10 -5 0 5 10 15
Pitch, θny (°)

-30

-15

0

15

30

45
15

Predicted Pitch, θny (°)

Conclusions
In this article, we showed how optimal deep learning can
give highly accurate models for robot touch using tactile
images from optical tactile sensors. To illustrate this
application, we considered local pose estimation of a 3D
surface or edge in contact with the BRL tactile fingertip
(TacTip), a biomimetic optical tactile sensor. To predict
the pose accurately, the trained network should be insensitive to how the sensor has reached its current pose. We
developed models that were insensitive to motion-dependent shear by including representative examples of these
motions as unlabeled perturbations of the training data.
However, these made it difficult to predict the pose, to
the extent that improperly tuned models gave extremely
suboptimal results. For a systematic approach, we introduced Bayesian optimization of the network and training
hyperparameters to find the most accurate models.
Consequently, the models were highly accurate (e.g.,
a 0.1-mm depth and 0.3º surface orientation) even though
the data were randomly perturbed by an unknown shearing motion. The models were also robust to the surface
shape and texture, as demonstrated by using the predicted poses to control a robot sliding the sensor across complex 3D objects (Figure 1).
Overall, the approach for robot touch introduced here
offers the potential for safe and precise physical interaction
with complex environments, encompassing tasks from
exploring natural objects to closed-loop dexterous manipulation. Even though we used the TacTip optical tactile sensor, a
similar approach should apply to other high-accuracy tactile
sensors, such as the GelSight, provided that they can slide
repeatedly across objects without damage. This work aims to
bring artificial tactile sensing one step closer to human performance and so raises the question of whether humans use
similar strategies during their own tactile interactions. In our
view, soft tactile sensors such as human fingertips cannot
function usefully in natural environments unless they have a

Predicted Yaw, θnz (°)

The gain matrices K p, K i were diagonal with proportional
gains of 0.5 and integral gains of 0.3 and 0.1 for translations and
rotations, respectively. The error e was evaluated between the
predicted pose and a reference normal to the surface or edge.
Using the PoseNet models (see the "Optical Tactile Sensing for Deep Learning" section) for tactile pose estimation of
a 3D surface and 3D edge (discussed in the "Tactile Pose Estimation for a 3D Surface" section) resulted in successful object
exploration across two complex 3D objects: surface exploration across a porcelain bust and edge-following around a container top (Figure 1). The exploration encompassed a region
bounded by the robot's reach and a joint-space singularity
across the bust (on the right cheek). The estimated pose is
shown in the insets, with the surface normals in red and edge
normals in blue along the trajectories.

MAE = 6.4°

t=0

-45
-45 -30 -15 0 15 30 45
Yaw, θnz (°)

t

Ts (t) = K p e (t) + K i / e (t l ).

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

75
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
