IEEE Technology and Society Magazine - March 2018 - 66

often do not appear in later ones, especially when the
later ones have greater statistical power and are careful-
ly designed to test the prediction (rather than discover
new phenomena).
Norton [16] and Feest [17] offer independent but
similar discussions of replication disputes, that is,
cases in which there is a scientific dispute over wheth-
er a result has been replicated. Both Norton and Feest
show that such disputes frequently turn on whether (or
which) follow-up experiments are epistemologically
relevant to the original experiment, that is, whether
there is reason to think that both experiments produce
evidence concerning the same results. If the follow-up
experiment is relevant, and the phenomenon fails to
appear, then it is at least uncertain whether or not the
original result was correct; thus, proponents of the
original experiment (often the researchers who con-
ducted it) will often argue that the follow-up experi-
ment had design errors that made it irrelevant to the
original experiment. In terms of extrapolation, there is
no reason to expect a phenomenon in situation A to
extrapolate to situation B if the two situations are sub-
stantially different.

It is practically impossible to
demonstrate the safety of AVs based
on real world crash rates until long
after they are in widespread use.
To return to the cookie example, suppose a rival psy-
chologist attempts to replicate the original experiment
using humanities students, rather than engineers, and
chocolate and ginger candy rather than cookies. The
rival psychologist finds no evidence that students prefer
chocolate. The first psychologist might claim that this
experiment was not relevant to her original version: dif-
ferent majors have different cultures, so students in dif-
ferent majors may not have the same preferences; and
candy preferences are not necessarily the same as
cookie preferences. So, the first psychologist con-
cludes, the "replication" is not actually a replication of
her original experiment at all, and her result still stands.
But her rival maintains that these are not substantial
differences, and concludes that the original result has
been refuted.
To see this in a real-world dispute, consider the sci-
entific response to [12], which reported that an attempt
to replicate 100 high-profile psychological studies was

66

largely unsuccessful: depending on exactly how suc-
cesses were counted, as few as 36% of the studies
were successfully replicated. In reply, [18] argued that
"many of OSC's replication studies used procedures
that differed from the original study's procedures in
substantial ways." For example, "an original study that
gave younger children the difficult task of locating tar-
gets on a large screen was replicated by giving older
children the easier task of locating targets on a small
screen." Different researchers might disagree about
whether age makes a difference. One group of psychol-
ogists might argue that the experimental task is well
within the abilities of children at both ages; these
researchers would presumably regard the second
experiment as substantially the same as the first, at
least in terms of the research participants. Another
group of psychologists might argue that older children
will be much better at the task than younger children;
these researchers would presumably regard the second
experiment as substantially different, and therefore
irrelevant as a replication attempt.
More generally, both [16] and [17] point out that
debates over the relevance of follow-up experiments -
whether or not the two situations are substantially
similar or different - involve substantial controver-
sies over how the result is supposed to work. What
counts as a substantial difference depends on the
causal story that's behind the purported result. In
other words, the "design errors" alleged in follow-up
experiments are often not methodological or formal
statistical problems  - such as contaminated experi-
mental materials or incorrect data analysis - but
instead depend on substantive views about the phe-
nomenon in question, such as substantive disagree-
ments about the age at which children are capable of
performing certain kinds of perceptual tasks. The
controversy may not be settled by simply appealing to
"the evidence" because the two sides may not be able
to agree on which experiments count as evidence.
One side can always dismiss the other side's replica-
tion attempt by finding a subtle difference between
the two experiments.
What does all this have to do with AVs? Above, we
saw that it is practically impossible to demonstrate the
safety of AVs based on real-world crash rates until long
after they are in widespread use. This led to the sugges-
tion of using experiments, simulations, or formal
proofs. However, the challenges involved in extrapolat-
ing from one experiment to another also appear in
extrapolating from experimental results to real-world
driving conditions. Is a test track experiment, offline
simulation, or "fleet learning" human-AV comparison
analysis, relevant to real-world driving conditions? This
depends on substantive views about which factors are

IEEE Technology and Society Magazine

∕

march 2018



Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2018

Contents
IEEE Technology and Society Magazine - March 2018 - Cover1
IEEE Technology and Society Magazine - March 2018 - Cover2
IEEE Technology and Society Magazine - March 2018 - 1
IEEE Technology and Society Magazine - March 2018 - Contents
IEEE Technology and Society Magazine - March 2018 - 3
IEEE Technology and Society Magazine - March 2018 - 4
IEEE Technology and Society Magazine - March 2018 - 5
IEEE Technology and Society Magazine - March 2018 - 6
IEEE Technology and Society Magazine - March 2018 - 7
IEEE Technology and Society Magazine - March 2018 - 8
IEEE Technology and Society Magazine - March 2018 - 9
IEEE Technology and Society Magazine - March 2018 - 10
IEEE Technology and Society Magazine - March 2018 - 11
IEEE Technology and Society Magazine - March 2018 - 12
IEEE Technology and Society Magazine - March 2018 - 13
IEEE Technology and Society Magazine - March 2018 - 14
IEEE Technology and Society Magazine - March 2018 - 15
IEEE Technology and Society Magazine - March 2018 - 16
IEEE Technology and Society Magazine - March 2018 - 17
IEEE Technology and Society Magazine - March 2018 - 18
IEEE Technology and Society Magazine - March 2018 - 19
IEEE Technology and Society Magazine - March 2018 - 20
IEEE Technology and Society Magazine - March 2018 - 21
IEEE Technology and Society Magazine - March 2018 - 22
IEEE Technology and Society Magazine - March 2018 - 23
IEEE Technology and Society Magazine - March 2018 - 24
IEEE Technology and Society Magazine - March 2018 - 25
IEEE Technology and Society Magazine - March 2018 - 26
IEEE Technology and Society Magazine - March 2018 - 27
IEEE Technology and Society Magazine - March 2018 - 28
IEEE Technology and Society Magazine - March 2018 - 29
IEEE Technology and Society Magazine - March 2018 - 30
IEEE Technology and Society Magazine - March 2018 - 31
IEEE Technology and Society Magazine - March 2018 - 32
IEEE Technology and Society Magazine - March 2018 - 33
IEEE Technology and Society Magazine - March 2018 - 34
IEEE Technology and Society Magazine - March 2018 - 35
IEEE Technology and Society Magazine - March 2018 - 36
IEEE Technology and Society Magazine - March 2018 - 37
IEEE Technology and Society Magazine - March 2018 - 38
IEEE Technology and Society Magazine - March 2018 - 39
IEEE Technology and Society Magazine - March 2018 - 40
IEEE Technology and Society Magazine - March 2018 - 41
IEEE Technology and Society Magazine - March 2018 - 42
IEEE Technology and Society Magazine - March 2018 - 43
IEEE Technology and Society Magazine - March 2018 - 44
IEEE Technology and Society Magazine - March 2018 - 45
IEEE Technology and Society Magazine - March 2018 - 46
IEEE Technology and Society Magazine - March 2018 - 47
IEEE Technology and Society Magazine - March 2018 - 48
IEEE Technology and Society Magazine - March 2018 - 49
IEEE Technology and Society Magazine - March 2018 - 50
IEEE Technology and Society Magazine - March 2018 - 51
IEEE Technology and Society Magazine - March 2018 - 52
IEEE Technology and Society Magazine - March 2018 - 53
IEEE Technology and Society Magazine - March 2018 - 54
IEEE Technology and Society Magazine - March 2018 - 55
IEEE Technology and Society Magazine - March 2018 - 56
IEEE Technology and Society Magazine - March 2018 - 57
IEEE Technology and Society Magazine - March 2018 - 58
IEEE Technology and Society Magazine - March 2018 - 59
IEEE Technology and Society Magazine - March 2018 - 60
IEEE Technology and Society Magazine - March 2018 - 61
IEEE Technology and Society Magazine - March 2018 - 62
IEEE Technology and Society Magazine - March 2018 - 63
IEEE Technology and Society Magazine - March 2018 - 64
IEEE Technology and Society Magazine - March 2018 - 65
IEEE Technology and Society Magazine - March 2018 - 66
IEEE Technology and Society Magazine - March 2018 - 67
IEEE Technology and Society Magazine - March 2018 - 68
IEEE Technology and Society Magazine - March 2018 - 69
IEEE Technology and Society Magazine - March 2018 - 70
IEEE Technology and Society Magazine - March 2018 - 71
IEEE Technology and Society Magazine - March 2018 - 72
IEEE Technology and Society Magazine - March 2018 - 73
IEEE Technology and Society Magazine - March 2018 - 74
IEEE Technology and Society Magazine - March 2018 - 75
IEEE Technology and Society Magazine - March 2018 - 76
IEEE Technology and Society Magazine - March 2018 - 77
IEEE Technology and Society Magazine - March 2018 - 78
IEEE Technology and Society Magazine - March 2018 - 79
IEEE Technology and Society Magazine - March 2018 - 80
IEEE Technology and Society Magazine - March 2018 - 81
IEEE Technology and Society Magazine - March 2018 - 82
IEEE Technology and Society Magazine - March 2018 - 83
IEEE Technology and Society Magazine - March 2018 - 84
IEEE Technology and Society Magazine - March 2018 - 85
IEEE Technology and Society Magazine - March 2018 - 86
IEEE Technology and Society Magazine - March 2018 - 87
IEEE Technology and Society Magazine - March 2018 - 88
IEEE Technology and Society Magazine - March 2018 - 89
IEEE Technology and Society Magazine - March 2018 - 90
IEEE Technology and Society Magazine - March 2018 - 91
IEEE Technology and Society Magazine - March 2018 - 92
IEEE Technology and Society Magazine - March 2018 - Cover3
IEEE Technology and Society Magazine - March 2018 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com