IEEE Robotics & Automation Magazine - June 2019 - 85

Intelligent Robot Instruction Experiments
and the geometric reasoning aspects are useful in both work
Life skills that require object manipulation and discrimination and personal life.
were taught in these experiments. In our setting, the student
Instruction strategies appropriate to the skill being taught
and the IRI stand across from each other at a table during were selected. For the geometric assembly skill instruction, an
instruction, as seen in Figure 6. The IRI autonomously per- SMP prompting strategy was used with backward chaining,
forms the instruction, prompting, observation, evaluation, along with a multiple-baseline-across-skills experimental
and feedback (correction or reinforcement) loop shown in
Figure 3. No WoZ techniques were used for these experiments. The complete IRI system was implemented as a suite
of C++ and Python software modules, leveraging the Robot
Operating System (ROS; http://ros.org) for messaging, interprocess communication, and common robotics libraries.
The custom object-tracking system shown in Figure 7
provided the IRI with the ability to observe objects with
which both the student and robot interacted and thus accurately interpret the student's performance. The object-tracking system was implemented as a set of custom ROS nodes
using OpenCV to process live image streams from a camera
mounted under a transparent tabletop, as illustrated in
Figure 8. Details of the vision system performance are Figure 6. The interaction setting for instruction.
reported in [15], where we previously introduced response
prompting for instruction with an IRI. Interaction and feedback are provided through synthesized speech, speech recognition, and gestures.
The robotic hardware for this research is a Meka Robotics
M3 mobile humanoid robot [Figure 8(a)] with 7 degrees-offreedom (DoF) arms, 5-DoF hands, and a sensor head with
2-DoF movement. For this research, the IRI makes use of one
PrimeSense short-range (version 1.09) camera, one universal
serial bus camera, a Bluetooth microphone, and stereo speakers. The robot is equipped with two personal computers-one
providing real-time functionality of the base, arms, hands,
and lift and the second dedicated to the vision and audio
components.
Two skills were taught by the IRI:
Figure 7. The object tracker graphical user interface, with live,
1) making change, i.e., given a dollar and a purchase price, adjustable parameters (upper left) and the annotated live image
(lower left). The magnified image on the right shows an enlarged
first use a calculator to determine how much change is view of the annotated image. The annotations include position,
due, then present the IRI with the correct change in orientation, size, centroid location, and bounding box for each object.
coins (originally presented in [15]
and summarized in the following)
2) geometric assembly of larger objects
from smaller pieces, as shown in Figure 9, for which prerequisite skills
were taught using AR instruction
Transparent
(see the "Intelligent AR Instruction
Table Top
Experiments" section).
Transparent
Both skills involved live interaction
Answer Box
Table Top
with objects and demonstration of corE-Stop
rect responses.
Diffuse
Making change is a life and vocaHD Camera
Lighting
tional skill that involves calculating the
HD
Diffuse Lighting
correct quantities and denominations
Camera
of currency to be exchanged after a
(a)
(b)
cash transaction. The assembly skills
taught in our geometric assembly Figure 8. The table setup for the instructional setting from the (a) student's view and (b)
experiment are useful in job settings, overhead. HD: high-definition.
JUNE 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://www.ros.org

IEEE Robotics & Automation Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2019