IEEE Technology and Society Magazine - March 2018 - 65

it may be reasonable to assume that the human acted
to prevent a crash; these data can be examined to
determine whether the AV system also anticipated the
crash and whether, if the human hadn't intervened, the
system also would have attempted to avoid the crash.
Overall, this strategy allows the collection of AV-relevant
data even when the AV system is not actually control-
ling the vehicle.
However, experiment- or simulation-based methods
have their own limitations. Insofar as an experimental
situation is different from real-world conditions, it's not
obvious that we can extrapolate from the experiment
to the real world. For instance, a comparison to coun-
terfactual AV system behavior using fleet learning data
could tell us whether the AV system would have
attempted to avoid an anticipated crash, but it does
not tell us whether, say, icy road conditions would have
led to a crash anyways. A crash (or avoiding a crash) is
the result of a complex interaction between the driver,
the vehicle, and the environment. Substituting a
human driver for an AI system in simulation can tell us
what a different driver might have done differently, but
not how the vehicle and environment would have
responded differently.
Another approach, inspired by methods used in soft-
ware validation and control theory, would be to develop
formal, mathematical proofs that a given system is safe.
For example, Claire Tomlin and her coauthors derive
safety criteria for mathematical representations of
mechanical systems [10] such as traveling distances
between vehicles [11]. Tomlin's methods identify
"reachable states" of the given system, then develop
control laws - the rules that the AV system uses to
determine how to accelerate or brake or steer, for
example - that guarantee that the reachable states are
a subset of the "safe" states. Consider a platoon of
automated trucks - several automated trucks travel-
ing, one immediately behind the other, to allow drafting
for fuel efficiency. Tomlin's approach might consider AV
system reaction times and cargo mass to calculate
maximum deceleration rates for a following truck.
These maximum deceleration rates could be used to
determine the minimum distance between platoon
trucks such that a following truck can safely stop if the
lead truck abruptly stops. Importantly, the "can" here is
guaranteed by a mathematical proof.
However, these formal methods depend on mathe-
matical assumptions, and generally tell us nothing
about what happens when the assumptions are not sat-
isfied. Again, we may not be able to extrapolate from a
mathematical model to a real-world situation. To under-
stand the challenges of extrapolation, let us consider a
high-profile case where extrapolation has failed: the rep-
lication crisis.
march 2018

∕

Using the example of climate
science/policy, we see that political
controversy mightn't be settled by
simply appealing to "what science
tells us" - even when science
speaks with a single clear voice.
Replication and Extrapolation
The replication crisis has caused a great deal of conster-
nation across many fields of scientific research. Most
prominently this has occurred in psychology, but also
neuroscience, econometrics, and cancer research [12]-
[15]. In these fields, experimental research findings
have been much more difficult to reproduce than ex-
pected, both methodologically - due to a lack of details
about study or analysis methods, for example - and
statistically - after re-running the experiment and
crunching the numbers, the previously-reported effects
often do not appear, or are much more ambiguous.
Replication - and replication failure - can be
understood in terms of prediction and extrapolation.
The data gathered in a given experiment directly tell us
what happened only in that particular experimental
run. Researchers routinely make an extrapolative pre-
diction: that the result observed in this particular
experimental run will also appear in other, similar
experimental runs. A replication can then be under-
stood as an attempt to check this prediction, that is,
investigating whether or not the result observed in
experiment A will indeed appear in experiment B. Sup-
pose a psychologist observes in an experiment that a
group of engineering students pick chocolate chip cook-
ies much more often than gingersnaps. She attempts to
replicate this finding, making the prediction that that
the same result will be observed in a new experiment.
With this prediction, she extrapolates from the first
experiment to the second.
Sometimes, purely because of chance, the prediction
will turn out to be false. Suppose it just so happens
that, on the second experiment, the psychologist got a
sample of engineers who love gingersnaps. This kind of
replication failure is not worrisome; it can be avoided
with larger sample sizes and detected statistically with
enough replications of the experiment.
By contrast, the replication crisis is that the predic-
tion has turned out to be false much more frequently
than expected. Results observed in one experiment run

IEEE Technology and Society Magazine

65



Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - March 2018

Contents
IEEE Technology and Society Magazine - March 2018 - Cover1
IEEE Technology and Society Magazine - March 2018 - Cover2
IEEE Technology and Society Magazine - March 2018 - 1
IEEE Technology and Society Magazine - March 2018 - Contents
IEEE Technology and Society Magazine - March 2018 - 3
IEEE Technology and Society Magazine - March 2018 - 4
IEEE Technology and Society Magazine - March 2018 - 5
IEEE Technology and Society Magazine - March 2018 - 6
IEEE Technology and Society Magazine - March 2018 - 7
IEEE Technology and Society Magazine - March 2018 - 8
IEEE Technology and Society Magazine - March 2018 - 9
IEEE Technology and Society Magazine - March 2018 - 10
IEEE Technology and Society Magazine - March 2018 - 11
IEEE Technology and Society Magazine - March 2018 - 12
IEEE Technology and Society Magazine - March 2018 - 13
IEEE Technology and Society Magazine - March 2018 - 14
IEEE Technology and Society Magazine - March 2018 - 15
IEEE Technology and Society Magazine - March 2018 - 16
IEEE Technology and Society Magazine - March 2018 - 17
IEEE Technology and Society Magazine - March 2018 - 18
IEEE Technology and Society Magazine - March 2018 - 19
IEEE Technology and Society Magazine - March 2018 - 20
IEEE Technology and Society Magazine - March 2018 - 21
IEEE Technology and Society Magazine - March 2018 - 22
IEEE Technology and Society Magazine - March 2018 - 23
IEEE Technology and Society Magazine - March 2018 - 24
IEEE Technology and Society Magazine - March 2018 - 25
IEEE Technology and Society Magazine - March 2018 - 26
IEEE Technology and Society Magazine - March 2018 - 27
IEEE Technology and Society Magazine - March 2018 - 28
IEEE Technology and Society Magazine - March 2018 - 29
IEEE Technology and Society Magazine - March 2018 - 30
IEEE Technology and Society Magazine - March 2018 - 31
IEEE Technology and Society Magazine - March 2018 - 32
IEEE Technology and Society Magazine - March 2018 - 33
IEEE Technology and Society Magazine - March 2018 - 34
IEEE Technology and Society Magazine - March 2018 - 35
IEEE Technology and Society Magazine - March 2018 - 36
IEEE Technology and Society Magazine - March 2018 - 37
IEEE Technology and Society Magazine - March 2018 - 38
IEEE Technology and Society Magazine - March 2018 - 39
IEEE Technology and Society Magazine - March 2018 - 40
IEEE Technology and Society Magazine - March 2018 - 41
IEEE Technology and Society Magazine - March 2018 - 42
IEEE Technology and Society Magazine - March 2018 - 43
IEEE Technology and Society Magazine - March 2018 - 44
IEEE Technology and Society Magazine - March 2018 - 45
IEEE Technology and Society Magazine - March 2018 - 46
IEEE Technology and Society Magazine - March 2018 - 47
IEEE Technology and Society Magazine - March 2018 - 48
IEEE Technology and Society Magazine - March 2018 - 49
IEEE Technology and Society Magazine - March 2018 - 50
IEEE Technology and Society Magazine - March 2018 - 51
IEEE Technology and Society Magazine - March 2018 - 52
IEEE Technology and Society Magazine - March 2018 - 53
IEEE Technology and Society Magazine - March 2018 - 54
IEEE Technology and Society Magazine - March 2018 - 55
IEEE Technology and Society Magazine - March 2018 - 56
IEEE Technology and Society Magazine - March 2018 - 57
IEEE Technology and Society Magazine - March 2018 - 58
IEEE Technology and Society Magazine - March 2018 - 59
IEEE Technology and Society Magazine - March 2018 - 60
IEEE Technology and Society Magazine - March 2018 - 61
IEEE Technology and Society Magazine - March 2018 - 62
IEEE Technology and Society Magazine - March 2018 - 63
IEEE Technology and Society Magazine - March 2018 - 64
IEEE Technology and Society Magazine - March 2018 - 65
IEEE Technology and Society Magazine - March 2018 - 66
IEEE Technology and Society Magazine - March 2018 - 67
IEEE Technology and Society Magazine - March 2018 - 68
IEEE Technology and Society Magazine - March 2018 - 69
IEEE Technology and Society Magazine - March 2018 - 70
IEEE Technology and Society Magazine - March 2018 - 71
IEEE Technology and Society Magazine - March 2018 - 72
IEEE Technology and Society Magazine - March 2018 - 73
IEEE Technology and Society Magazine - March 2018 - 74
IEEE Technology and Society Magazine - March 2018 - 75
IEEE Technology and Society Magazine - March 2018 - 76
IEEE Technology and Society Magazine - March 2018 - 77
IEEE Technology and Society Magazine - March 2018 - 78
IEEE Technology and Society Magazine - March 2018 - 79
IEEE Technology and Society Magazine - March 2018 - 80
IEEE Technology and Society Magazine - March 2018 - 81
IEEE Technology and Society Magazine - March 2018 - 82
IEEE Technology and Society Magazine - March 2018 - 83
IEEE Technology and Society Magazine - March 2018 - 84
IEEE Technology and Society Magazine - March 2018 - 85
IEEE Technology and Society Magazine - March 2018 - 86
IEEE Technology and Society Magazine - March 2018 - 87
IEEE Technology and Society Magazine - March 2018 - 88
IEEE Technology and Society Magazine - March 2018 - 89
IEEE Technology and Society Magazine - March 2018 - 90
IEEE Technology and Society Magazine - March 2018 - 91
IEEE Technology and Society Magazine - March 2018 - 92
IEEE Technology and Society Magazine - March 2018 - Cover3
IEEE Technology and Society Magazine - March 2018 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com