IEEE Robotics & Automation Magazine - September 2019 - 11

ETHICAL, LEGAL, AND SOCIETAL ISSUES

Unintended Consequences of Biased
Robotic and Artificial Intelligence Systems
By Ludovic Righetti, Raj Madhavan, and Raja Chatila

The IEEE Robotics and Automation Research and Practice Ethics Committee (RARPEC)
is intended as a platform to exchange ideas and discuss the impacts and practice
of robotics and automation (R&A) technologies in research, development, and
deployment that appear to pose ethical questions for humanity. With increased
awareness and controversies surrounding R&A, RARPEC is publishing a series of
opinion pieces that will focus on separating hype from reality by providing an objective
and balanced treatment of technological, ethical, legal, and societal perspectives. Third
in the series, this piece focuses on the implications and consequences of bias in robotic
systems. Please send your feedback and suggestions to the chair of the committee, Raj
Madhavan, at raj.madhavan@ieee.org. We look forward to your comments!

t seems natural to assume that wellengineered robots and artificial intelligence (AI)-based systems will
always behave in an unbiased and
nondiscriminatory manner when they
interact with or provide services to
people. As long as engineers are not
malicious, won't robots perform tasks
equally well, independent of one's
gender, ethnicity, age, or socioeconomic
background?
While this could be considered a
common-sense assumption, a growing
body of work shows that algorithms
often exhibit unintended and unexpected biases, leading to unfair discrimination against certain groups of
people. Although seldom addressed in
robotics, these issues will certainly
permeate all applications that involve
interactions with humans and have
recently raised concerns in the machinelearning community. In this article, we
discuss some of these issues, including
how they could impact robotics and
automation and potential approaches

Digital Object Identifier 10.1109/MRA.2019.2926996
Date of publication: 5 September 2019

to ensure the fair deployment of robotics technologies.
When Algorithms Reinforce or
Lead to Discrimination
Commercial computer-vision systems
are prime examples of systems that
unfairly discriminate or contribute to
reinforcing stereotypes due to misclassification errors. In 2015, it was reported
that Google's photo service classified
black people as gorillas in certain pictures, reinforcing a racist stereotype.
Surprisingly, the problem appeared difficult to resolve: more than two years
later, the seemingly preferred solution
was to simply remove the gorilla category from the classification algorithm
[1]. An audit of several commercial
facial-recognition systems for gender
classification revealed that every tested
system performed systematically worse
with women and, in particular, with
dark-skinned women as compared with
light-skinned men [2]. Amazon's Rekognition system had 30% more classification errors for dark-skinned women
while close to perfect classification
performance for light-skinned men.

Interestingly, the audit, which publicly
named some of the tested companies,
had a positive impact. Several months
later, those companies had released new
versions of the algorithms that demonstrated a significant decrease in performance disparities. These examples shed
light on how seemingly value-free technologies can lead to racial or genderbased inequities, as technology-based
services may not function equally well
for everyone. This finding raises serious
concerns because facial recognition is
being widely deployed, particularly for
government services and law-enforcement applications.
However, the problem goes beyond
mere computer vision. For example,
Amazon reportedly developed a hiring
tool to help rank candidates using data
from previous hires [10]. The system
was shown to systematically downgrade
candidates who attended all-women
colleges (and, generally, all resumes
containing the word women). Despite
efforts to remove such bias, it seems
that the hiring tool was finally abandoned. Interestingly, gender-based discrimination appears to be difficult to
prevent due, among other causes, to a
history of gender-biased hiring practices that permeate the data. This illustrates how an algorithm can potentially
introduce or reinforce, and indefinitely
perpetuate, already rampant discriminatory practices.
A more disconcerting example is
the infamous COMPAS algorithm that
certain courts in the United States
used to assess the likelihood that a
criminal defendant becomes a recidivist.

SEPTEMBER 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2019