IEEE Technology and Society Magazine - March 2022 - 70

Feature
algorithms that are able to detect patterns within
data [7]. Often these data-derived patterns are then
used to make automated decisions or to engage in
predictions. These systems characteristically are
able to improve their performance over time-or
" learn " on particular tasks-by analyzing additional
data. A major requirement for machine-learning is
the availability of data-typically large amounts
of data-that can be analyzed for useful patterns
[7]. The type and quality of the data supplied to
a machine learning system is therefore crucial to
its functionality.
One of the most important sources of embedded
values in legal AI systems come from biases in data.
Of major concern: if there are biases in the underlying
data used to build a machine learning system,
this can lead to biases in how the system performs.
These biases may be hard to detect but can produce
undesirable, unlawful, or unfair outputs.
For example, as discussed, some U.S. courts use
computer models in parole or sentencing decisions
for criminal defendants [13]. These systems attempt
to predict (among other things) the risk that criminal
defendants will commit future crimes. Such predictions
are often be based upon general crime data
and purport to identify correlations of future criminal
behavior. However, often these data sets are heavily
based upon recorded police activity. Notably, if the
police activity data upon which the machine-learning
model was built was itself biased, the predicted
outcome might be similarly skewed. For example,
imagine that police tended to patrol disproportionally
in minority neighborhoods compared to other
neighborhoods (thereby skewing the base rate of
police interactions with minority populations compared
to the populate generally), or if police tended
to arrest members of minority groups at a higher
rate than nonminority groups, all other things being
equal, for similar behaviors. Such biased choices on
the part of the police to patrol or arrest at a disproportionate
base rate would be reflected in the police
recorded data. In other words, the recorded police
data would not reflect an objective model of actual
crime activity, but would reflect skews introduced by
police behavior-patrol, arrest, and data recording
practices. This in turn would create a misleadingly
high correlation between minority status and risk of
offending, which might be embedded into a system
analyzing patterns in such police data [13]. Similarly,
machine-learning models can detect seemingly
70
neutral correlates-such as zip code-that can
become proxies for characteristics that are unlawful
to consider, such as race-that might be incorporated
by the system.
Importantly, it often is hard to detect when a
machine learning system has such biases in its
data model because they can become embedded
computationally in subtle ways. While such
data-induced biases are of concern in computing
generally, they are particularly problematic in legal
systems given the substantial real-world effects that
they can have on the legal rights of individuals. To
the extent that such systems are used in the application
of law, designers should be particularly attentive
to the possibility that the data upon which their
model was built might be systematically skewed in
some way that might preference or inhibit some
social subgroups.
Inscrutable and uninterpretable models
Some artificial intelligence models are difficult
to interpret and understand. This idea goes by various
names, including the " intelligibility, " " comprehensibility, "
or " inscrutability " problems [13],
[17]. The general idea is the following: When we
create a computer system, we often need to know
why the system made a particular decision that it
did. However, while all systems encode their decision-making
processes in some sort of computer
model, some models are more understandable by
people than others. For example, certain rules-based
artificial intelligence systems can provide a logical,
step-by-step analysis, in human-readable form, as to
why they took the particular decision that they did.
Similarly, human written source code tends to be
comparatively easy to understand, as programmers
can systematically proceed step by step through the
instructions to understand why a piece of software
acts the way that it does.
By contrast, some machine-learning techniques
produce extremely complex computer models
whose underlying logic can be very difficult, if not
impossible, for humans to inspect and comprehend.
For example, neural networks-particularly deep
learning neutral network systems-have proved very
adept at automating certain tasks. Notably, however,
neural networks tend to encode their patterns in
models that are notoriously difficult to understand.
In many cases, neural networks are able to produce
highly accurate results on complex tasks using an
IEEE Technology and Society Magazine
IEEE Technology and Society Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2022
