IEEE Technology and Society Magazine - December 2019 - 22

who struggle to gain unbiased infor-
mation in a timely fashion because
autonomous weapons could make
decisions quickly and more accu-
rately. Indeed, a parallel argument
led to automated defensive weap-
ons systems that exist today like
the Patriot missile system or Israel's
Iron Dome though these systems
are primarily automated systems,
not autonomous systems, and their
accuracy has been debated. It has
been recognized that humans sim-
ply cannot often respond in time to
threats like incoming rocket attacks
since they have an inherent limita-
tion known as the neuromuscular
lag [9]. Even if a person is paying
attention perfectly to an evolving sit-
uation, there is still an approximate
half second lag for a person to see
a problem and then act accordingly.
This is why automated and autono-
mous defensive systems are given
the role and ability to authorize
weapons launch.
Autonomous offensive weapons
are given a goal (e.g., find and kill the
enemy) and then execute the mis-
sions by themselves until the goal is
accomplished. Currently there are no
offensive autonomous weapons used
in actual dynamic battlefield scenari-
os. Automated offensive weapons do
exist and are routinely used, such as
the combination of drones carrying
missiles like the Air Force's Predator.
One of my jobs as an F/A-18 pilot was
to get a radar-guided missile as close
as possible to an airborne target (like
an enemy helicopter) without being
attacked. Once I was close enough
to give a missile an initial guess as
to the target's location, I would shoot
the missile, and then once the mis-
sile had a radar lock on the target,
run away to get out of the enemy's
missile envelope. Drones like the
Predator and Reaper do this exact
same job, but they leave the human
out of the cockpit, and supervising
from a safe distance. However, the

22

to reasoning are extremely brittle
and often unable to cope with even
the smallest perturbation of data.
Given these fundamental problems
in current computer vision technol-
ogy that make it very unreliable in
understanding the world, especially
in dynamic situations, any weapon
system that requires autonomous
reasoning based on machine learn-
ing, either offensive or defensive,
will be deeply flawed. Moreover, it
is entirely likely that these computer
vision weaknesses could be exploit-
ed and create new cybersecurity
concerns, which have already been
demonstrated in street sign [6] and
face recognition [14] applications.
Because of these technological limi-
tations, at the current time, the role of
acquiring a target is still very much a
function for humans to execute.
From a pure engineering assess-
ment approach, any autonomous
system that currently relies on com-
puter vision systems to reason about
dynamic environments is likely to
be extremely unreliable, especially
in situations never before encoun-
tered by the system. Unfortunately,
this is exactly the nature of warfare.
Thus, maintaining human control in
current military operations is criti-
cal, since the technology is so deep-
ly flawed. But while the technology
currently cannot cope with uncer-
tainty on the battlefield, it is far too
premature to say that will always be
the case. As the Operation Provide
Comfort example illustrates, while
humans are better than current
deep learning algorithms at consis-
tent target identification, they are
very much error prone in real world
scenarios. As will be discussed in
a later section, if autonomous tar-
geting systems in dynamic settings
could be shown to be superior to
humans at some point in the future,
which is a very high bar, I argue that
not only should we use them, but we
have an obligation to do so.

fundamental roles of the humans
and computers do not change in
drone missions. Thus, in current oper-
ations, a human guides a drone to
the target, identifies the target, and
approves weapons launch.
While the role of target identi-
fication is currently assigned to
humans, the U.S. military has spent
a significant amount of money over
the past 20 or more years to develop
various forms of automated target
recognition systems for unmapped
and dynamic targets, but progress
has been slow and difficult. Recent
approaches have tried to add more
advanced forms of computer vision
and machine learning techniques to
improve performance, but resulting
systems have had high false posi-
tive rates and other technical prob-
lems [13]. The limitations of machine
learning and computer vision sys-
tems are the Achilles' heel for all
autonomous systems, military and
civilian. Computer vision systems
struggle, especially under dynamic
and unfamiliar conditions, to inte-
grate incoming sensor data into the
"brain" of an autonomous system to
form a world model by which action
decisions can be made [3]. Despite
the hype surrounding the advance-
ment of driverless cars that lever-
age artificial intelligence in their
computer vision systems, they and
all such autonomous systems are
deeply flawed in their ability to reli-
ably identify objects [4], [10].
Figure 1 illustrates just how com-
puter vision flaws using a deep learning
algorithm are manifested in identi-
fying vehicles in different poses. In
all three examples, a typical road
vehicle (school bus, motor scooter,
firetruck) is shown in a normal pose,
with 3 other unusual poses. While a
bus on its side may be a rare oc-
currence in everyday driving, such
unusual poses are part of the typi-
cal battlefield environment. Deep
learning data-driven approaches
IEEE TECHNOLOGY AND SOCIETY MAGAZINE

∕

DECEMBER 2019
IEEE Technology and Society Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2019
