IEEE Technology and Society Magazine - December 2019 - 21

on MHC tasks that have high likeli-
hoods of human error with deadly
unintended consequences.
Fundamentally this question of
meaningful control aligns with a
question faced every day by con-
trols engineers who must determine
which functions and roles automa-
tion/autonomy should have versus
what should be allocated to humans
when designing a complex system.
To answer such questions, engi-
neers and designers must know
the strengths and limitations of
humans as well as of the technolo-
gies, and such boundaries are not
always clear. It is often not clear
whether a technology or a human
should or could be in control from a
design standpoint, such as whether
self-driving cars are actually better
than humans across a number of
driving scenarios [4]. Such design
gaps inevitably lead to discussions
of ethical and social impacts of the
technology, especially in safety criti-
cal systems like transportation and
medical systems, as well as autono-
mous weapons.
To shed more light on these is-
sues, two vignettes of actual mili-
tary engagements are provided to
frame the MHC debate, which is
really a discussion about role alloca-
tion between humans and autono-
mous systems. I suggest that what is
needed is not a call for meaningful
human control of autonomous weap-
ons but rather a focus on meaningful
human certification of such systems.

The original target was a Yugo-
slav supply warehouse, about
500 yards from coordinates
were provided to the Air Force
by the U.S. Central Intelligence
Agency, in part due to faulty
maps and a flawed review pro-
cess that failed to catch the
error [12].

Operation Provide Comfort
In 1994, the United States
sought to stop Iraqi attacks
on Kurds in northern Iraq by
enforcing a no-fly zone through
Operation Provide Comfort.
The no-fly zone was patrolled
by U.S. Air Force F-15 fighters,
supported by an A i r b o r n e
Warning and Control System
(AWACS) aircraft. On April 14,
two U.S. Army Black Hawk
helicopters transported U.S.,
French, British and Turkish
commanders, as well as Kurd-
ish personnel across this
zone when the F-15 fighters
misidentified them as Iraqi
Hind helicopters. The fighters
launched air-to-air missiles,
killing all 26 onboard the
Black Hawks. One person on
the AWACS plane gave the
Black Hawks permission to
enter the no-fly zone. Howev-
er, due to a communication
breakdown, this information
was not shared with another
controller on the same AWACS
communicating with the F-15s.
One government accident
investigation found 130 caus-
al human errors for this single
accident [8].
These two case studies exemplify
the kinds of missions faced by mili-
tary operators in deciding when and
where to launch an airborne weap-
on with likely lethal consequenc-
es. The Chinese Embassy incident
represents what should have been
a relatively low uncertainty engage-

The Chinese Embassy
Bombing
During NATO's Operations Allied
Force campaign in Yugosla-
via in May of 1999, the Chinese
embassy in Belgrade was acci-
dentally bombed with a U.S. Air
Force GPS-guided Joint Direct
Attack Munition (JDAM). Three
Chinese journalists were killed,
with more than 20 people injured.
DECEMBER 2019

∕

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

ment given the intended target was
static and known, while the Operation Provide Comfort case was one
of high uncertainty with a dynamic
and unknown potential threat. Un-
derstanding the relative level of
uncertainty is key to understanding
how much human control could and
should be exerted in these scenari-
os, and what the implications are for
the use of autonomous weapons in
these settings [3].

High Uncertainty
Engagements Are Outside
the Realm of Autonomous
Weapons - For Now
Unfortunately, the problems sur-
rounding the Operation Provide
Comfort incident are not new to the
military and even today are sadly all
too common. There are many very
similar incidents where a suspected
target emerges on or near a battle-
field, information about this target
is misinterpreted, and because of
the "fog of war" which includes in-
complete information, psychological
biases, and human error, a weapon
is launched, killing innocent civil-
ians or friendly forces [12].
One of the biggest problems with
high uncertainty targeting situa-
tions is time pressure. Identification
and classification of such targets is
always the highest priority, but when
targets are moving and perceived
as a threat, there is limited time for
those humans in control (the F-15
pilots in the Provide Comfort exam-
ple) to make a high-quality decision.
Research has shown that warfight-
ers under time pressure in the air
and on the ground are predisposed
to a number of psychological bias-
es that cause them not to evaluate
all relevant pieces of information,
which in many cases leads to disas-
trous outcomes [2], [11].
Autonomous weapons could, in
theory, aid human decision makers in
these time-critical targeting scenarios

21
IEEE Technology and Society Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2019
