IEEE Robotics & Automation Magazine - March 2013 - 36

(a)

(b)

(c)

(d)

(e)

Figure 10. Henry fetching a towel from his kitchen with the PR2. (a) Grasping a cabinet handle. (b) Pushing open a cabinet door.
(c) Opening a drawer. (d) Grasping a towel inside the drawer. (e) Navigating to the desired dropoff location.

control [Figure 8 (c)], highlighting the benefits of autonomous assistance.
However, our experiences in Henry's home also highlight
the importance of providing tools for direct control when
autonomy is either not applicable or fails outright. For
instance, in an initial test, it was discovered that the robot's fan
causes curtains to billow and register as obstacles in the
robot's navigation map, which stops autonomous navigation.
Also, during the towel-fetching task, a synchronization error
between the robot's Laser Rangefinder and its computers
caused most of the autonomous navigation attempts to fail. In
both cases, the direct control was used to compensate. In
other situations, autonomy is simply not applicable. For
instance, Henry used the robot's forearm to push shut the cabinet door by directly controlling the gripper, a task for which
no autonomous tool was available.
Like the remote training for the manipulation around the
body tasks, we also found that training was a key enabler for
using the object manipulation tools. In this case, a simulation
environment allowed Henry to practice complex tasks in the
comfort of his home. Even though the simulator cannot accurately capture the complexity of real-life situations, it can still
help the operator become familiar with the interface and
robot with no risk of injury or damage. We believe that both
appropriate training mechanisms, and interfaces that provide
a range of tools with varying levels of autonomous assistance,
are key to allowing the assistive robots to successfully perform
ADLs in real homes. More information about the Interactive
Manipulation interface can be found in [4].
An Alternative Interface Method: Head Tracking
Although the Interactive Manipulation interface is effective,
we would also like the opportunity to free Henry from always
needing a computer monitor in front of him. With this in
mind, we have begun experiments with a head-tracking system for contextual interactions with the robot such as selecting an object for manipulation in the real world. Using the
system in [5] with the Kinect sensor, we track the 3-D position and orientation of Henry's head in real time and use this
36

IEEE ROBOTICS & AUTOMATION MAGAZINE

March 2013

as a pointer for objects in the world. This is analogous to the
Clickable World interface in [13]. We compute a vector normal to the coronal plane of the head and then intersect this
with the robot's 3-D world model. This intersection point can
be used to determine what Henry is looking at. A full overview and initial results are described in [11].
While initial experiments with this system have been
encouraging, they have also highlighted a number of areas
that need improvement. The tracked head pose is noisy, so we
use a mean filter to trade off stability of the pose estimate
against responsiveness. When making large head movements,
responsiveness seems to be more important; when trying to
dwell on an object, stability is more important. This implies
that we need to make the tracker settings either more intelligent or more controllable.
To be useful as a pointing device, the system needs to offer
feedback about what it thinks Henry is pointing at. In our initial prototype, we fixed a laser pointer to the robot's head for
this purpose, but this is not a good long-term solution unless
Henry can control when the laser is on.
Finally, we intend to make the sensor itself less obtrusive,
either by using sensors that are already in widespread use
(such as a Kinect mounted on a TV) or by mounting the sensor on the robot itself in a way that allows it to track Henry's
head without interfering with normal robot operations.
Assistance with Social Interaction
Moving beyond manipulation tasks, we are using the PR2 to
support EADLs such as socializing [15]. Henry has expressed
frustration with the slow, inexpressive methods of communication available to him, which leave him an outsider in many
conversations. The communication board that Henry and
Jane use (Figure 11) requires the conversation partner to infer
the gaze direction of the user to spell words [19], a task that
requires patience and skill. Augmentative communication
systems [1] such as text-to-speech (TTS) generators are functional but frustratingly slow.
In contrast to these methods, AMMs have the distinct
advantages of embodiment and physical presence. This has

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2013