IEEE Technology and Society Magazine - December 2019 - 43

study, because we were curious to see how Autonomous
Weapon Systems as future technology would compare on
this aspect to the current technology of human operated
drones, but in retrospect anxiety is an emotion instead of
a moral value. Therefore, this study should be regarded as
exploratory and requires follow-up studies in different cultures to see if the results will hold.

Recommendations for Further Research
Given the exploratory nature of this study and the limitations mentioned above, we suggest several recommendations for further research. The first is to run
follow-up studies to investigate if the results on moral
values related to Autonomous Weapon Systems hold in
different samples and in different cultures that have a
different attitude toward drones, robots, and AI technology than we as the authors in the Western world have.
Another approach to consider which values are relevant in the deployment of Autonomous Weapon Systems
is the specification of values into design requirements to
see how these values would be perceived in practice.
This can be made visible by means of a value hierarchy
[67], which is a hierarchical structure of values, norms,
and design requirements. A value hierarchy will to make
the value judgements that are required for the translation
explicit, transparent, and debatable. To do so, the values
that are described in the natural language will need to be
translated to "formal values in a formal language" [68].
One way of formalizing values into norms would be to
use a convention of rules represented as: "X counts as Y"
or "X counts as Y in context C" [69]. The explicitness of
values in formal rules and visibility in a value hierarchy
would allow for critical reflection in debates and pinpoint
the value judgements that are disagreed on.
In this study we provided empirical results to fill the
knowledge gap on moral values related to Autonomous
Weapon Systems that aim to understand the perception
of people regarding the introduction of Autonomous
Weapon Systems. The recommendations for further
research, being a follow-up study for the moral values in
related AI fields, and the creation of a value hierarchy,
will be conducted at our research group at Delft University of Technology by our graduate students. Continuing
our research effort in this field would help better define
the "fundamental moral line" that should not be crossed
[33]. The results of these studies reveal a common
ground regarding the moral values of human dignity and
anxiety, pertaining to the introduction of Autonomous
Weapon Systems. The concerns for Autonomous Weapon
Systems mentioned in the public debate deserve to be
carefully considered in future debates on the ethics of
the design and deployment of Autonomous Weapon Systems. We believe that our empirical results substantiate
some of the views and opinions on Autonomous Weapon
DECEMBER 2019

∕

Systems affecting human responsibility in the current discourse; these results could be used to move forward in
the ethical debate on Autonomous Weapon Systems.

Acknowledgment
The authors would like to thank Sydney Levine, Iyad
Rahwan of the Scalable Cooperation Group at the Media
Lab of M.I.T., and Jeroen van der Hoven of Delft University of Technology for their insightful comments, sharp
discussions, and supervision of the research.

Author Information
Ilse Verdiesen and Filippo Santoni de Sio are with Delft
University of Technology, Jaffalaan 5, 2628 BX Delft, The
Netherlands. Virginia Dignum is with Delft University of
Technology, Jaffalaan 5, 2628 BX Delft, The Netherlands,
and with Umeå University, 901 87 Umeå, Sweden. Email:
e.p.verdiesen@tudelft.nl.

References

[1] S.J. Russell and P. Norvig, Artificial Intelligence: A Modern
Approach. Malaysia: Pearson, 2016.
[2] J.J. Bryson, "Patiency is not a virtue: AI and the design of ethical
systems," in 2016 AAAI Spring Symposium Series, 2016.
[3] L. Floridi and J.W. Sanders, "On the morality of artificial agents,"
Minds and machines, vol. 14, no. 3, pp. 349-379, 2004.
[4] H.M. Roff, "Weapons autonomy is rocketing," Foreign Policy,
Sept. 28, 2016; http://foreignpolicy.com/2016/09/28/weapons-autonomyis-rocketing/.
[5] U.S. Air Force, "Unconscious US F-16 pilot saved by Auto-pilot,"
YouTube, Sept. 15, 2016; https://www.youtube.com/watch?v=
RQBFJkMnBcA, accessed Feb. 19, 2017.
[6] J. Carpenter, Culture and Human-Robot Interaction in Militarized Spaces: A War Story. Taylor & Francis, 2016.
[7] C. Heyns, "Report of the Special Rapporteur on Extrajudicial,
Summary, and Arbitrary Execution, United Nations Human Rights
Council," United Nations 23rd Session, vol. 9, Apr. 2013.
[8] Campaign to Stop Killer Robots, "Campaign to Stop Killer
Robots," 2018; https://www.stopkillerrobots.org/, accessed July 15,
2017.
[9] Office of the High Commissioner, "A/HRC/31/66, Joint report of
the Special Rapporteur on the rights to freedom of peaceful assembly and of association and the Special Rapporteur on extrajudicial,
summary or arbitrary executions on the proper management of
assemblies," United Nations, 2016.
[10] J. Kaag and W. Kaufman, "Military frameworks: Technological
know-how and the legitimization of warfare," Cambridge Rev. Int.
Affairs, vol. 22, no. 4, pp. 585-606, 2009.
[11] M. Rosenberg and J. Markoff, "The Pentagon's 'Terminator
Conundrum': Robots that could kill on their own," New York
Times, 2016.
[12] B. Burridge, "UAVs and the dawn of post‐modern warfare: A perspective on recent operations," RUSI J., vol. 148, no. 5, pp. 18-23, 2003.
[13] P. Asaro, "How just could a robot war be," Current Issues In
Computing and Philosophy, pp. 50-64, 2008.
[14] A. Krishnan, "Killer robots," Legality and Ethicality of Autonomous Weapons. Farnham, U.K.: Ashgate, 2009.
[15] M. Guarini and P. Bello, "Robotic warfare: Some challenges in moving from noncivilian to civilian theaters," Robot Ethics: The Ethical
and Social Implications of Robotics, vol. 129, p. 136, 2012.
[16] N. Sharkey, "Automated killers and the computing profession,"
Computer, vol. 40, no. 11, 2007.
[17] N. Sharkey, "Killing made easy: From joysticks to politics,"
Robot Ethics: The Ethical and Social Implications of Robotics,
pp. 111-128, 2012.

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

43


https://foreignpolicy.com/2016/09/28/weapons-autonomyis-rocketing/ https://foreignpolicy.com/2016/09/28/weapons-autonomyis-rocketing/ https://www.youtube.com/watch?v=RQBFJkMnBcA https://www.youtube.com/watch?v=RQBFJkMnBcA https://www.stopkillerrobots.org/

IEEE Technology and Society Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2019

Contents
IEEE Technology and Society Magazine - December 2019 - Cover1
IEEE Technology and Society Magazine - December 2019 - Cover2
IEEE Technology and Society Magazine - December 2019 - 1
IEEE Technology and Society Magazine - December 2019 - Contents
IEEE Technology and Society Magazine - December 2019 - 3
IEEE Technology and Society Magazine - December 2019 - 4
IEEE Technology and Society Magazine - December 2019 - 5
IEEE Technology and Society Magazine - December 2019 - 6
IEEE Technology and Society Magazine - December 2019 - 7
IEEE Technology and Society Magazine - December 2019 - 8
IEEE Technology and Society Magazine - December 2019 - 9
IEEE Technology and Society Magazine - December 2019 - 10
IEEE Technology and Society Magazine - December 2019 - 11
IEEE Technology and Society Magazine - December 2019 - 12
IEEE Technology and Society Magazine - December 2019 - 13
IEEE Technology and Society Magazine - December 2019 - 14
IEEE Technology and Society Magazine - December 2019 - 15
IEEE Technology and Society Magazine - December 2019 - 16
IEEE Technology and Society Magazine - December 2019 - 17
IEEE Technology and Society Magazine - December 2019 - 18
IEEE Technology and Society Magazine - December 2019 - 19
IEEE Technology and Society Magazine - December 2019 - 20
IEEE Technology and Society Magazine - December 2019 - 21
IEEE Technology and Society Magazine - December 2019 - 22
IEEE Technology and Society Magazine - December 2019 - 23
IEEE Technology and Society Magazine - December 2019 - 24
IEEE Technology and Society Magazine - December 2019 - 25
IEEE Technology and Society Magazine - December 2019 - 26
IEEE Technology and Society Magazine - December 2019 - 27
IEEE Technology and Society Magazine - December 2019 - 28
IEEE Technology and Society Magazine - December 2019 - 29
IEEE Technology and Society Magazine - December 2019 - 30
IEEE Technology and Society Magazine - December 2019 - 31
IEEE Technology and Society Magazine - December 2019 - 32
IEEE Technology and Society Magazine - December 2019 - 33
IEEE Technology and Society Magazine - December 2019 - 34
IEEE Technology and Society Magazine - December 2019 - 35
IEEE Technology and Society Magazine - December 2019 - 36
IEEE Technology and Society Magazine - December 2019 - 37
IEEE Technology and Society Magazine - December 2019 - 38
IEEE Technology and Society Magazine - December 2019 - 39
IEEE Technology and Society Magazine - December 2019 - 40
IEEE Technology and Society Magazine - December 2019 - 41
IEEE Technology and Society Magazine - December 2019 - 42
IEEE Technology and Society Magazine - December 2019 - 43
IEEE Technology and Society Magazine - December 2019 - 44
IEEE Technology and Society Magazine - December 2019 - 45
IEEE Technology and Society Magazine - December 2019 - 46
IEEE Technology and Society Magazine - December 2019 - 47
IEEE Technology and Society Magazine - December 2019 - 48
IEEE Technology and Society Magazine - December 2019 - 49
IEEE Technology and Society Magazine - December 2019 - 50
IEEE Technology and Society Magazine - December 2019 - 51
IEEE Technology and Society Magazine - December 2019 - 52
IEEE Technology and Society Magazine - December 2019 - 53
IEEE Technology and Society Magazine - December 2019 - 54
IEEE Technology and Society Magazine - December 2019 - 55
IEEE Technology and Society Magazine - December 2019 - 56
IEEE Technology and Society Magazine - December 2019 - 57
IEEE Technology and Society Magazine - December 2019 - 58
IEEE Technology and Society Magazine - December 2019 - 59
IEEE Technology and Society Magazine - December 2019 - 60
IEEE Technology and Society Magazine - December 2019 - 61
IEEE Technology and Society Magazine - December 2019 - 62
IEEE Technology and Society Magazine - December 2019 - 63
IEEE Technology and Society Magazine - December 2019 - 64
IEEE Technology and Society Magazine - December 2019 - 65
IEEE Technology and Society Magazine - December 2019 - 66
IEEE Technology and Society Magazine - December 2019 - 67
IEEE Technology and Society Magazine - December 2019 - 68
IEEE Technology and Society Magazine - December 2019 - 69
IEEE Technology and Society Magazine - December 2019 - 70
IEEE Technology and Society Magazine - December 2019 - 71
IEEE Technology and Society Magazine - December 2019 - 72
IEEE Technology and Society Magazine - December 2019 - 73
IEEE Technology and Society Magazine - December 2019 - 74
IEEE Technology and Society Magazine - December 2019 - 75
IEEE Technology and Society Magazine - December 2019 - 76
IEEE Technology and Society Magazine - December 2019 - 77
IEEE Technology and Society Magazine - December 2019 - 78
IEEE Technology and Society Magazine - December 2019 - 79
IEEE Technology and Society Magazine - December 2019 - 80
IEEE Technology and Society Magazine - December 2019 - 81
IEEE Technology and Society Magazine - December 2019 - 82
IEEE Technology and Society Magazine - December 2019 - 83
IEEE Technology and Society Magazine - December 2019 - 84
IEEE Technology and Society Magazine - December 2019 - Cover3
IEEE Technology and Society Magazine - December 2019 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com