IEEE Technology and Society Magazine - September 2019 - 57

E

ngineers and scientists often seek to
use their work to benefit society, but
they face difficulty in tracing the benefits of research and development. Assessing the technical and societal outcomes of research and development is
an old and continuous challenge [1][5]. Bozeman and Sarewitz [4] argue that formal economic calculations - like tracking patent and publications
counts - cannot fully capture the impact that a technological development brings. Instead, they argue for "public value mapping:" causally "mapping" specific research
and development activities to public goods by employing
social scientists to conduct historical case studies of
those innovation areas. Case studies and qualitative research in general thus present a way to both understand
the benefits of past research and find the sources of innovation and the causal links between how research and
development connects to public goods.
Historic case studies of research and development
(R&D) can guide engineers on how to study innovation
and cultivate it in new work. The year 2019 marks the
50th anniversary of the most famous case study on the
sources of innovation: the 1960s Project Hindsight. An
initiative from the U.S. Department of Defense (DOD), it
involved case studies of 20 major weapons systems
developed from the end of World War II to 1962. For
each system, a team of roughly five analysts examined
innovation events to understand:
the sources of innovation used in the initial development or block upgrades of weapons systems at the
DOD; and
■ the extent to which investments in R&D helped to
improve weapons systems overall and whether a metric could estimate the cost effectiveness of research.
■

In science policy conversations today, Project Hindsight is known for its controversial claim that basic and
applied science contribute little to innovation. Its analysis did not consider events 20 years before an innovation - effectively employing a "time horizon" on the
study of innovation. Many have critiqued Hindsight, in
particular for its use of this 20-year time horizon. The
strongest critique of Hindsight came from a study commissioned by the United States National Sciences Foundation (NSF) in response: TRACES.
Using a policy and technical analysis, we will assess
both TRACES and Hindsight to show that Project Hindsight has value beyond its controversial claim and that
methodological differences undercut the criticism of
Hindsight's time horizon. To do so, we will first review
the history of the Hindsight report and examine ensuing
political controversies. We will then review Hindsight's

SEPTEMBER 2019

∕

methods and summarize its results. In contrast, we will
present the methods and results of the TRACES counter-study. Taken together with their criticisms, these
reports offer guidance on the broad array of ways to
study the sources of innovation today. We hope that our
initial analysis of these reports can be elaborated on in
the future by historians and other policy researchers.
Our primary goal here it to contribute to current science policy research by helping make practitioners
aware of these important milestones and to highlight
the role qualitative case study research can play in government decision-making. Case studies like these, especially if paired with a connection to public values (as
Bozeman and Sarewitz [4] define it), could empirically
demonstrate the links between research, innovation,
and public good, and potentially guide new policies to
better achieve public value. In this sense, public values
can involve pursuing societally agreed upon goals. Public value mapping case studies explore how actions
using science, technology, engineering, and math
(STEM) research achieve those goals. Hindsight's focus
on weapon system development thus fits an interest in
obtaining only a subset of broader public values, specifically focusing on national security values. The methods
discussed here could apply to many areas outside of
national security, thus helping engineers learn how to
better contribute to public welfare.

Project Hindsight Summary
Hindsight conducted an analysis of contemporary history by examining the sources of innovation in key weapon systems. Sherwin Chalmers, the Deputy Director of
the DOD Office of the Director of Defense Research and
Engineering, inspired the Hindsight study [6]-[8]. Raymond Isenson, an engineer with significant experience
planning DOD programs, became the director of Project
Hindsight [9].1 Contemporaneous news articles about
Hindsight do not mention that Secretary of Defense
Robert McNamara's more analytical approach to management likely motivated the study.2 Our tentative
impression from reading the early documents is that the
DOD wanted to get better information to help it manage
its system developments, and may not have expected
the controversy that later emerged.
1

It is worth noting the scientific background of the two authors, Sherwin Chalmers (1916-1998) and Raymond Isenson (1920 - 2009). Both men had Ph.D.'s in
physics. Chalmers' career returned to science after his stint at the DOD. Isenson
continued for a few years to write on topics related to technical management.
2
In reviewing this paper, Alex MacDonald questioned whether Hindsight was
an intentional effort to explore if the linear model of innovation is true or
not. This could have been part of a generational reassessment of Vannevar
Bush's 1945 report "Science: The Endless Frontier" [13]. We have not found
confirming evidence in the Hindsight reports that the study's intent or methods were explicitly responding to Vannevar Bush or if they specifically had a
conception of the linear model. Narayanamurti and Odumosu's 2016 book
Cycles of Innovation and Discovery provides tangible examples of how
the linear model is used in DC policy discourse to harmful effects.

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

57
IEEE Technology and Society Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2019
