IEEE Technology and Society Magazine - September 2019 - 65

TABLE 5. Summary Comparison of TRACES and Hindsight Research Event Classifications.
TRACES Classifications
Hindsight Classifications

Non-Mission

Science

Pure or applied math

Validation of theory

Combined theory or experiment

Development of a new material

Conception or demonstration of a new concept

Technology

Analysis or measurement of material performance
Engineering

Mission

Development

Demonstration of a capability to perform a given function

Development of manufacturing technique

Development of a complete system

*possible interpretation

including five criticisms of the Project Hindsight RXD
event framework, while generally ignoring TRACES. The
most important framework critique is about the problem
of equal priority bias. While there were only a handful of
RXD events attributed to basic science in Hindsight, one
of those developments was the creation of the transistor.
The transistor was incredibly important in establishing
computers, which changed the construction and capability of weapons system. Still, the Hindsight report does
note that not all RXD events were of equal value. Nevertheless, the complaint of equal priority bias is fair in its
contribution to the controversy of the report's findings.
Despite this critique, the data on RXD event timing and
details remains of likely importance. Kreilkamp also suggests that Hindsight should have more deeply involved
historians and management scholars.
A critique might also question the effect of poor
human memory in leading people to forget earlier sources of innovation. Research projects can span decades
and a specific system may go through several changes in
team members during development [5]. Given long time
horizons and the frailty of human memory, recollection
may not provide a reliable account. However, Hindsight's
use of historical archive data and interviews with many of
those involved reasonably mitigates this uncertainty. The
distribution of research events, as those shown in Figure 1, appears plausible, with the peak of events occurring several years before a system development.
Additional research could complement memory challenges present in Hindsight-like work. Real-time data collection, including embedded ethnographers, may more fully
assess the sources of a system's innovation. Indeed,
many engineers only fully assess the value dimensions of
their own decisions when confronted in the moment by
questions from an embedded observer [17].

SEPTEMBER 2019

∕

It is worth noting that government agencies do
research to support studies of innovation, but structured case studies like Hindsight are relatively rare. U.S.
government agencies do support real-time data collection, as with NASA's Cost Analysis Data Requirement
(CADRe). Qualitative case studies on emerging projects
could augment such systems. NASA employs a Knowledge Management Officer to record lessons learned
from space flight efforts. NASA's History office also
funds historical research on space flight projects. No
NASA equivalent to the Project Hindsight study has
been performed as far as we are aware.10

On Repeating Hindsight
Why has a systematic study like Hindsight not been
repeated? While the Hindsight methodology is not
perfect, its rich data and insights dull its sharpest
criticisms. The academic literature has not rigorously explored the accuracy of Hindsight as compared
to TRACES.
Cost may be a reason why no Hindsight-like study
has been attempted, but that seems unlikely. Hindsight took approximately 40 worker years to complete. Assuming a $150 thousand cost per person year
(the average fully loaded cost today for a government
employee), the study may have cost about 6 million
2019 dollars phased over its three years. Government
agencies occasionally spend this amount of money to
provide strategic advice. A much smaller version of Hindsight could focus on one or a few case studies, instead
of the 20 that Hindsight assessed.

10
One of the authors is a NASA civil servant, who has focused on Project
Hindsight as a way to study methods that NASA might better employ in the
future to guide innovation.

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

IEEE Technology and Society Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2019