IEEE Technology and Society Magazine - June 2019 - 43

"Models of Threat" and an "Ethics of Care."3 It will first
outline these concepts in broad terms. It will then
examine two illustrative cases, in the area of predictive
policing, which epitomize each approach. It concludes
with some observations and reflections on how to
design better and more ethical AI through an Ethics of
Care approach.
Perhaps the greatest ethical concerns over algorithmic decisions have been raised around the use of datadriven algorithms in policing, law enforcement, and
judicial proceedings. One well-researched and much
discussed example from the Florida judicial system
involves the use of algorithms to predict future recidivism in convicts as a basis for determining the length of
their sentences.4 Another growing application is predictive policing - the use of historic crime data to identify
individuals or geographic areas with elevated risks for
future crimes, in order to target them for increased
policing. Predictive policing has been controversial - as
it aspires to prevent crime, it also raises questions of
prejudice and precrime and effectively treating individuals and communities as guilty of (future) crimes for acts
they have not yet committed and may never commit
[21], [22]. This central controversy of prejudice and precrime is amplified and exacerbated by more general
concerns over the implicit biases contained in historic
data sets, and the obvious implications for racial, gendered, ethnic, religious, class, age, disability, and other
forms of discriminatory policing.
Predictive policing as a term can refer to a variety of
technologies and practices. The technical usage of the
term usually refers to algorithmic processes for predicting locations or individuals with high probabilities
of being involved in future crime, based upon historical data patterns [23]. Recent approaches utilize "big
data" techniques and arguably entail forms of mass
3
Neither term is original, and each is meant to evoke traditions of thought
and their general perspective, while not necessarily implying that the specific projects described were conscious of, or directly influenced by, those
traditions. "Threat Modeling" has been an important methodology in cybersecurity for identifying, assessing, prioritizing, and mitigating threats and
vulnerabilities since at least the early 2000s [14], while "Threat Perception"
has been a key concept in international relations and political psychology in assessing military threats and deterrence strategies [15]. "Ethics of
Care" has been gaining popularity in medical and educational ethics since
its introduction by Carol Gilligan to explain moral development in child psychology in the late 1970s and its extension by Nel Noddings into a moral
theory based on interpersonal relationships of caregiving and receiving in
the early 1980s [16].
4
In an analysis of 7000 sentencing cases in Broward County, Florida, over
the period 2012-2013 that used the COMPAS software, journalists found
similar error rates in the assessment and sentencing of white and black
convicts, but diametrically opposed in their direction. White convicts were
more likely to be erroneously predicted not to commit future crimes, while
black convicts were more likely to be erroneously predicted to commit
future crimes, resulting in shorter sentences for white convicts and longer
sentences for black convicts [17].
Another study of the same dataset shows that amateur humans are
able to make better predictions than the COMPAS software, using the same
six factors as the software, and even better predictions can be made using
just two factors - defendant's age and number of past convictions [18].

JUNE 2019

∕

surveillance of the public [24]. However, these recent
algorithmic techniques and applications have their
roots in much older practices of collecting and utilizing
comparative statistics (better known as CompStat)
about crimes to manage large police forces, which
began in New York City in 1995. While many CompStat
programs utilized computer programs to calculate the
statistics from crime and accident reports and arrest
records and in some cases automatically generate
"pin-maps" of crime activity, CompStat was really a set
of data collection, analysis, and management practices
rather than a piece of software [25]. And CompStat has
seen its share of criticism, including from former
police officers [26].
Moreover, the algorithmic techniques that are
increasingly being employed by police forces draw
upon data that goes well beyond the digitized crime
reports of the CompStat legacy, or automatically generated "heat maps" of areas of high crime activity.5 In
recent years, police departments have begun deploying
and integrating large-scale video surveillance systems,
traffic cameras, license-plate and face recognition
technologies, audio gun-shot locators, cellphone interceptors, aerial surveillance, and a host of other surveillance and data-collection technologies. As these
systems become networked and produce large amounts
of data, there is increased pressure to analyze, integrate, and utilize this data for improving law enforcement, which leads to increased reliance on automation
and algorithms for sorting and sifting through that data
and translating it into policing priorities and strategies.
As such, the term predictive policing can be taken to
refer to a broad class of algorithmic and data-driven
practices and software tools utilized by police forces.
Predictive policing is also a good example of how AI
might be deployed more generally, and the ethical challenges that may arise.
A general approach to AI ethics is characterized here
as an "Ethics of Care." Ethics of Care uses predictive
policing, and the design of AI-based systems within it,
to lay out the framework for an AI Ethics of Care. In particular we look at two recent, but very different, implementations of data-driven interventions on youth gun
violence in Chicago, Illinois, U.S.A. Predictive policing is
particularly good for this purpose for several reasons.
As should be clear from the discussion above, policing
is an area the gives rise to a number of critical ethical
and legal issues, and has relevance not only to society
at large, but to a host of other governmental functions
and other industries. It is also an area that has an historical practice of data collection, and recent trials in
5
Such "heat maps" have become ubiquitous in the age of big data, and are
even reproduced, albeit at lower resolution, on real estate websites such
Trulia.com [27].

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

43
http://www.Trulia.com
IEEE Technology and Society Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2019
