IEEE Robotics & Automation Magazine - September 2017 - 54

sensors to scan for anomalies that the static cameras might
have missed and using its actuators to interact with the environment in ways that a static camera cannot. The robot's limitation, however, is that it can occupy only one physical
location at a time.
Consider that, late one night, the robot is patrolling the
east corridor on its way to the elevator hallway. Suddenly, one
of the fixed cameras detects a person moving in the north
corridor. At this time of day, the north corridor has restricted
access, arousing suspicion that someone is trespassing.
Assuming this event is communicated to the robot across the
network, the robot could turn around and proceed directly to
that location. Alternatively, the robot could continue along to
inspect the elevator hallway, which is also an important room
in the building. This example illustrates the kind of relevant
Table 1. The challenges of surveillance
decision making.
Challenge

Explanation

Constrained resources

A robot has a finite operation
time and cannot visit all locations
instantaneously.

Urgency/priority

A trespassing event left
unaddressed for too long can
turn into a robbery.

Uncertainty about
event occurrences

It is unknown when, where, and
even if an event will occur.

Uncertainty in decision
consequences

There is no guarantee that the
robot will succeed in its actions,
e.g., thwarting the trespasser.

Uncertainty in the
sensor data

Imperfect detection methods
may yield false positives and
false negatives.

Coordination of
decisions

A robot team should handle
events in parallel, avoiding
redundancy.

Intermittent
communication

This can occur, for example,
when robots traverse large and
complex spaces with dead zones.

Robot Actuators

Environment Actuators
Actuation

Decision Making

Aggregation and
Filtering

HMI

Localization

Event Detection

Robot Sensors

Event Detection
Sensors

Figure 2. The modular design of our surveillance framework.

IEEE ROBOTICS & AUTOMATION MAGAZINE

SEPTEMBER 2017

decisions that a patrolling robot could face, given its current
status and the status of the surveillance system. The decision
whether to respond immediately to an event or to continue
patrolling should be made carefully and deliberately, since it
could compromise the security of the building.
Modular System Design for Decision Making
In general, a mobile security robot will experience a sequence
of decision making about where to go and what to do, as long
as it is operating in the environment and events are being
detected by the network. To increase the autonomy of the networked robotic system, planning methodologies should consider several relevant aspects within the decision-making
problem, as summarized in Table 1.
In addition to accommodating various decision-making
methodologies, an effective autonomous surveillance framework needs to deal with a wide range of heterogeneous sensors and actuators exchanging information in real time, e.g.,
differing robot platforms, lasers, cameras, microphones, and
speakers. Therefore, we propose a modular framework for
security inspection that divides the overall system into components and defines a set of interfaces for component interaction and communication. The system is versatile enough to
allow for adaptable reuse as well as the incorporation of new
functionalities (e.g., new sensor technologies).
Figure 2 diagrams our modular surveillance framework.
Apart from the heterogeneous sensor and actuator modules, a
human-machine interaction (HMI) module is included to
display information (e.g., detected events) to the operator, to
receive remote commands (e.g., sending a robot to a desired
position), and to produce audible signals from each robot in
the form of speech, whereby the robot can interact with people in the environment.
Detecting and Disseminating Events
Events, such as a person requiring assistance or an intrusion,
form the basis for all intelligent surveillance activities. In this
section, we describe where these events come from and how
they are automatically detected and represented in support of
effective robot planning. For illustrative purposes, we focus
our description on the trespassing event introduced in the
"Motivating Example" section.
Image Processing
The multicamera system requires live video acquisition and
transmission. High-resolution camera images need to be captured and received at a steady rate and reliably enough to
perform event detection. This involves high-bandwidth computation, balanced across several high-performance servers,
each processing the images in real time.
Our surveillance system integrates the technique proposed
in [19] for both detecting people as they move around within
parameters of the designated area and for sensing other
events, such as a request for assistance or trespassing. Other
image-processing algorithms could be plugged in to our system since the framework is flexible, requiring only that new

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2017