IEEE Technology and Society Magazine - September 2022 - 55

justice system that perpetuate the systemic violence
against our most vulnerable populations.
AI in the justice system finds its place in courts
where risk is assessed, from bail, sentencing, to
parole. The question that plagues our decision-makers:
if this individual is released by this court, will
they pose a threat or risk to the public? The decision-maker's
obligation to the public is onerous, and
the responsibility to protect public safety is a heavy
burden. The use of predictive analyses by way of AI
tools may provide a tangible result, while also delivering
speed, objectivity, and efficiency [4, p. 11]. For
decision-makers with long court dockets, who are
faced with difficult decisions on an hourly basis, AI
presents the relief the justice system is seeking.
To date, the use of AI has not arrived in Canadian
courtrooms, but the American experience has raised
red flags. With the rapid proliferation of pretrial risk
assessments across the United States, judicial decision-making
and the traditional deference given
to judicial discretion is under threat. Advocates in
favor of pretrial risk assessments point to the potential
elimination of individual cognitive biases held
by members of the judiciary. Ewert and critics of AI
have signaled the underlying issues with algorithmic
decision-making in risk assessment tools.
When testifying for Jeffrey Ewert, Dr. Hart did
not reject risk assessment tools outright. The Federal
Court surmised from this expert's evidence that
risk assessment tools' reliability could extend to the
dominant group, but left out cultural nuance and
history into consideration and that actuarial tests,
like the ones in issue, were susceptible to cross-cultural
biases [3, ¶ 28]. The Federal Court wrote, " The
conclusion from Dr. Hart's expert report is that the
actuarial tests are not sufficiently predictably reliable
for Aboriginals because of the cultural variance
or bias of the tests " [3, ¶ 41].
The concerns highlighted by the Federal Court
in Ewert are similar to concerns expressed regarding
applications of AI, namely, the reinforcement of
stereotypes, myths, systemic racism, and poverty,
the prevalence of vulnerable populations, and conversely
the omission of groups from data. In Oakland,
CA, USA, an algorithm developed by PredPol
Inc., designed to assist police forces to target drug
crime, repeatedly sent police officers to the same
overpoliced, poor, and racialized communities.
White and affluent neighborhoods were spared the
September 2022
repeated police intervention, despite equal rates of
drug use between the races [33, p. 125].
The application of AI tools in pretrial risk assessments
has also proved to be faulty. In 2016, ProPublica
found that the COMPAS risk assessment tool
used in Broward County, FL, USA, produced unreliable
predictions and significant racial disparities
between White and Black defendants [4, p. 12]. In
2020, the Pretrial Justice Institute (PJI), a former supporter
of AI as a mechanism for bail reform, stated
clearly: " Regardless of their science, brand or age,
these tools are derived from data reflecting structural
racism and institutional inequity that impact
our court and law enforcement policies and practices.
Use of that data deepens the inequity " [4, p.
13], [35]. The PJI has gone on to reveal flaws that
affect the reliability of AI pretrial risk assessment
tools across the United States [35] and to withdraw
its support of AI as a tool for bail reform.
The example of Hewlett Packard's facial recognition
software makes the broader point of the impact
of Black faces not being recognized by HP's MediaSmart
computer, an AI platform. The software was able
to identify white faces but could not pick up nuances
in black faces. When exposed, HP's press release
read that their AI technology was " built on standard
algorithms that measure the difference in intensity of
contrast between the eyes and the upper cheek and
nose " [36, p. 161]. The question became, how did
these " standard algorithms " leave out black faces?
Facial recognition software is currently used by policing
and surveillance agencies as a current application
of AI technologies, despite critiques of its frailties [4],
[6], [35]. The potential for overbreadth of its reporting
and conclusions, as well as its failure to consider
nuance and detail remain primary concerns.
Data-driven, algorithmic decision-making and
AI pose problems to our justice system that can be
foreseen. The examples that have been cited and the
case study in Ewert show that minority groups are
actively missed from a tool's application or the tool's
lack of appreciation for history or cultural variances.
Browne describes the HP example as " prototypical
whiteness " [36, p. 161] that whiteness is the standard
tool of measurement, while minority groups represent
the deviation. Benjamin articulates that " by
decoding the racial dimensions of technology and
the way in which different genres of humanity are
constructed in the process, we gain a keener sense
of the architecture of power " [21, p. 32].
55
IEEE Technology and Society Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2022
