Tech Briefs Magazine - November 2021 - 48

Robotics & Automation
Soft Robots Use Camera and Shadows to Sense
Human Touch
An off-the-shelf USB camera captures the shadows made by hand gestures on the robot's skin.
Cornell University, Ithaca, New York
esearchers have created a low-cost
method for soft, deformable robots
to detect a range of physical interactions,
from pats to punches to hugs, without
relying on touch at all. Instead, a USB
camera located inside the robot captures
the shadow movements of hand gestures
on the robot's skin and classifies them
with machine-learning software.
Touch is an important mode of communication
for most organisms but has
been virtually absent from human-robot
interaction. One of the reasons is that
full-body touch used to require a massive
number of sensors and was therefore
not practical to implement.
R
The ShadowSense technology originated
through work to develop inflatable
robots that could guide people to
safety during emergency evacuations.
Such a robot would need to be able to
communicate with humans in extreme
conditions and environments. Imagine a
robot physically leading someone down
a noisy, smoke-filled corridor by detecting
the pressure of the person's hand.
Rather than installing a large number
of contact sensors - which would add
weight and complex wiring to the robot
and would be difficult to embed in a
deforming skin - the team took a counterintuitive
approach. In order to gauge
touch, they looked to sight. By placing a
camera inside the robot, they can infer
how the person is touching it and what
the person's intent is just by looking at
the shadow images.
The prototype robot consists of a soft,
inflatable bladder of nylon skin that is
stretched around a cylindrical skeleton,
roughly four feet in height, that is
mounted on a mobile base. Under the
robot's skin is a USB camera that connects
to a laptop. The researchers developed
a neural-network-based algorithm
that uses previously recorded training
data to distinguish among six touch gestures
- touching with a palm, punching,
touching with two hands, hugging,
pointing, and not touching at all - with
an accuracy of 87.5 to 96%, depending
on the lighting.
The robot can be programmed to re -
spond to certain touches and gestures
such as rolling away or issuing a message
through a loudspeaker. And the robot's
skin has the potential to be turned into
an interactive screen. By collecting
enough data, a robot could be trained to
recognize an even wider vocabulary of
interactions, custom-tailored to fit the
robot's task.
The robot doesn't even have to be a
robot. ShadowSense technology can be
incorporated into other materials, such
as balloons, turning them into touchsensitive
devices. In the future, the re -
searchers will try using optical devices
such as lenses and mirrors to enable
additional form factors.
ShadowSense also offers privacy. If
the robot can only see a human in the
form of their shadow, it can detect
what the person is doing without taking
high-fidelity images of their ap -
pearance. This provides a physical filter
and protection, and provides psychological
comfort.
The ability to physically interact and
understand a person's movements and
moods could ultimately be just as important
to the person as it is to the robot.
For more information, contact Jeff Tyson
at jeff.tyson@cornell.edu; 607-793-5769.
Robot Locates and Retrieves Buried Items
This robotic arm fuses data from a camera and antenna to locate and retrieve items buried
under piles and completely out of view.
Massachusetts Institute of Technology, Cambridge
A
robotic system called RFusion is a
robotic arm with a camera and
radio frequency (RF) antenna attached
to its gripper. It fuses signals from the
antenna with visual input from the
camera to locate and retrieve an item,
even if the item is buried under a pile
and completely out of view. The
RFusion prototype relies on RFID tags,
which are cheap, battery-less tags that
can be stuck to an item and reflect signals
sent by an antenna. Because RF
signals can travel through most sur48
Cov
faces,
RFusion is able to locate a tagged
item within a pile.
Using machine learning, the robotic
arm automatically zeroes in on the object's
exact location, moves the items on top of
it, grasps the object, and verifies that it
picked up the right thing. The camera,
antenna, robotic arm, and AI are fully integrated,
so RFusion can work in any environment
without requiring a special setup.
RFusion could have applications such
as sorting through piles to fulfill orders
in a warehouse, identifying and instal l -
www.techbriefs.com
ToC
ing components in an auto manufacturing
plant, or helping an elderly individual
perform daily tasks in the home.
RFusion begins searching for an object
using its antenna, which bounces signals
off the RFID tag (like sunlight being
reflected off a mirror) to identify a spherical
area in which the tag is located. It
combines that sphere with the camera
input, which narrows down the object's
location; for instance, the item can't be
located on an area of a table that is empty.
Once the robot has a general idea of
Tech Briefs, November 2021

http://www.techbriefs.com http://info.hotims.com/79419-902

Tech Briefs Magazine - November 2021

Table of Contents for the Digital Edition of Tech Briefs Magazine - November 2021