IEEE Technology and Society Magazine - December 2021 - 45

understood and accepted
effectiveness. "
as
evidence
of
* Implementation of measurement practices that
are practically feasible and widely adopted by
practitioners.
* Transparency such that measurement methods
and results are " open to scrutiny by experts and
the general public " [38].
EAD's recommendations include the following.
* Governments fund and support the establishment
of ongoing benchmarking exercises and
facilitate the creation of data sets that can be
used for effectiveness evaluations.
* Creators of AI systems " pursue valid measures
of the effectiveness of their systems ... through
participation in benchmarking exercises or
through conducting single-system validation
exercises. "
* Creators describe the procedures and results of
testing: 1) in language that can be understood by
both experts and nonexperts and 2) openly for
examination by all stakeholders.
* Researchers define meaningful effectiveness metrics
while seeking input from all stakeholders.
* Governments and industry associates undertake
educational efforts to inform operators of AI
systems deployed in the legal system and those
affected by those systems.
ABA resolution 112 regarding ethical and legal
issues related to AI use in law practice
In August 2019, the ABA House of Delegates
adopted Resolution 112 (Report), which " urges
courts and lawyers to address the emerging ethical
and legal issues related to the usage of [AI] in the
practice of law " [21]. The Report identifies three
specific issues of AI use in law practice: " 1) bias,
explainability, and transparency of automated decisions
made by AI; 2) ethical and beneficial usage
of AI; and 3) controls and oversight of AI and the
vendors that provide AI " [21]. It says that courts
and lawyers " should address situations where
their usage of AI may be flawed or biased " [21]
(emphasis added).
The Report says that lawyers increasingly use AI
to " improve the efficiency and accuracy of legal
services offered to their clients " [21]. It calls for
raising the awareness of courts and lawyers of the
issues involved in using (and not using) AI. " Given
December 2021
that many lawyers are focused on detail and control
over their matter[s], it is easy to see why 'the
greater danger might very well be underutilization
of, rather than overreliance upon, artificial intelligence' "
[21].
The report called for the establishment of a working
group to " define guidelines for legal and ethical
AI usage and potentially develop a model standard
that could come to the ABA House of Delegates for
adoption " [21]. Subsequently, the ABA Section of
Science & Technology and ABA's Center for Professional
Responsibility established a working group,
which prepared a report and proposed resolution
(Guidelines Proposal). I was a member of this working
group. Persons outside of the working group
decided not to present the Guidelines Proposal to
the House of Delegates, however, and the working
group was asked instead to fashion it into a white
paper, which has not yet happened.
The Guidelines Proposal " urges courts and lawyers
to adopt policies and practices for the effective,
competent, transparent, and accountable
use of [AI] in the practice of law and the administration
of justice " [49]. It says that courts and
lawyers " should ... appropriately investigate the
effectiveness of AI to evaluate whether it is fit for
its intended purpose " [49]. It calls for the testing
of AI outcomes and says that " [l]awyers should
only rely on AI tools that are adequately tested to
detect bias " [49].
The Guidelines Proposal also calls for transparency
about the effectiveness of AI, saying that " [l]
awyers and courts need to know whether AI actually
works. " " Lawyers and courts should require
their AI operators to provide evaluations of
the AI that will permit the lawyers or courts to
determine if the AI is in fact effective for the
given task " [49] (emphasis added).
Mechanisms for indirect enforcement
of soft law to create a culture of
effectiveness testing
Gutierrez and Marchant [2] have identified
numerous mechanisms for the indirect enforcement
of soft law governance of AI. Internal measures
include corporate boards, ethics committees, ethics
officers, and ombudspersons. External measures
include supply chains, NGO monitors, auditing and
certification, trade associations, professional societies,
liability insurers, grantors and funding agencies,
45
IEEE Technology and Society Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2021
