Education Week - January 23, 2013 - 20

EDUCATION WEEK

JANUARY 23, 2013

www.edweek.org

▲

COMMENTARY
www.edweek.org/go/commentary

International Tests Reveal
Surprises at Home and Abroad
By Martin Carnoy
& Richard Rothstein

olicymakers and pundits raise
alarms whenever international
test results are announced. In
December, upon release of new
scores from the Trends in International Mathematics and
Science Study, or timss, U.S. Secretary of
Education Arne Duncan called them “unacceptable,” saying they “underscore the
urgency of accelerating achievement ...
and the need to close large and persistent
achievement gaps.”
It was no different a little over two years
ago, when the Program for International
Student Assessment, or pisa, released its
latest scores. Secretary Duncan said they
showed Americans “napping at the wheel.
... As disturbing as these national trends
are for America, enormous achievement
gaps among black and Hispanic students
portend even more trouble for the United

States in the years ahead.”
Such conclusions are oversimplified, frequently exaggerated, and misleading. They
ignore the complexity of testing and may
lead policymakers to pursue inappropriate
and even harmful reforms.
Both timss and pisa publish not only average national scores, but also a rich database from which analysts can disaggregate
scores by students’ socioeconomic characteristics. Examining these can lead to more
nuanced conclusions than those suggested
from average national scores alone.
Yet, for some reason, although timss
published average national results this
past December, it only released its underlying database last week. This puzzling
procedure ensures that commentators
draw quick but ill-informed interpretations. Analysis of the database takes time,
and headlines from the initial release are
codified before scholars can complete more
careful study.
Since the last pisa release in 2010 (of a
test given in 2010), we have been digging

deeper into its database and examining
older databases for timss and for our domestic National Assessment of Educational Progress. We concentrated on adolescent scores—8th graders on timss and
naep, 15-year-olds on pisa—in the United
States; three top-scoring countries (Canada,
Finland, and South Korea); three similar
postindustrial countries (France, Germany,
and the United Kingdom); seven American
states; and three Canadian provinces for
which trends are available.
The timss executive director deemed the
report of our findings helpful (without endorsing our analysis in detail), but officials
of pisa and of the U.S. Department of Education were harshly critical. We have posted
their criticism, and our response to it, online (www.edweek.org/go/PISAresponse).
Some of our conclusions are obvious; some
are counterintuitive or startling. Here are
a few:
• A larger proportion of students in
the United States is disadvantaged
than in any comparison country. The

What’s in a Name?

Rethinking the Notion of ‘Noncognitive’
By David T. Conley

name can matter a lot. When social science
researchers wanted to make a distinction between how students approached
different aspects of the learning process,
they coined the somewhat awkward term
“noncognitive” to distinguish attitudes,
beliefs, and attributes from content knowledge, which
they labeled “cognitive.” They applied their newly minted
term to identify everything that was not, in their view,
grounded in, or directly derived from, rational thought.
This distinction reflected the idea that one type of thinking formed the basis of knowing and recalling information, and that the other originated in beliefs, attitudes,
and feelings.
Perhaps it’s time to move beyond our current overly
cautious approach to measuring elements of the learning
process that extend beyond content knowledge. Perhaps
it’s time to think of noncognitive dimensions of learning
as forms of thinking, rather than as a process that does
not involve cognition.
Are we not observing a higher form of thinking when
we see students persist with difficult tasks, such as overcoming frustration; setting and achieving goals; seeking
help; working with others; and developing, managing,
and perceiving their sense of self-efficacy? Are these
qualities not at least as important as knowing how well
students recall information about the year in which the
Civil War began, or how to factor a polynomial? Might
what we observe when we look for noncognitive factors
be a more complex form of cognition—a result of executive functioning by the brain as it monitors and adjusts
to circumstances to accomplish specific aims and objec-

tives? In other words, might these behaviors be manifestations not of feelings, but of metacognition—the mind’s
ability to reflect on how effectively it is handling the
learning process as it is doing so?
During the past century, cognitive measures have ascended to the pinnacle of credibility, while noncognitive
factors have languished in relative obscurity. Consumers of content-knowledge tests assumed that noncognitive indicators lacked the methodological rigor that was

number increased rapidly over the last decade, while in comparison countries it did
not. Nonetheless, reading and mathematics achievement of lower-social-class U.S.
students improved substantially, while
achievement of similarly disadvantaged
students declined in countries to which the
United States is frequently unfavorably
compared.
Thus, while the reading achievement on
pisa of the lowest-social-class students in
the United States grew by more than 0.2
standard deviations from 2000 to 2009, it
fell by an even larger amount in Finland. In
math, U.S. students from the lowest social
class also gained substantially, while scores
of comparable Finnish students declined.
This is surprising because the proportion of
disadvantaged students in Finland also fell,
and we might expect this to make the task
of devoting resources to them easier.
Certainly, even for the lowest social class
of students, Finland’s scores remain higher
than ours, but examination of trends (perPAGE 23 >

fulfilling prophecy. They anointed cognitive measures
as “scientific” and pronounced noncognitve techniques
as incapable of meeting the technical standards they
had developed for content tests. Because experts judged
noncognitive methods against inappropriate standards,
all noncognitive approaches came to be like the guy or
girl who gets all dressed up for the party but never gets
asked to dance. Over time, the very name noncognitive
came to symbolize, among educators and policymakers,
that the information generated by the tool or instrument
would be of limited use and value to shape curriculum,
instruction, or program improvement, let alone be factored into accountability systems.
We might better describe what has previously fallen
under the label of noncognitive factors as “metacognitive
learning skills.” While metacognition is a broad term, I
apply it here to describe a subset of all its facets, just

“

Are we not observing a higher form of thinking when we see students
persist with difficult tasks, such as overcoming frustration?”
the hallmark of their cognitive siblings. Psychometricians dedicated considerable resources over the years
to improving cognitive measures, while they devoted
relatively less effort and energy to creating noncognitive
tools of comparable rigor.
Standardized-test developers created knowledge examinations that were assumed to be free of noncognitive
influences. The tests didn’t attempt to gauge the processes students used to learn. Over time, educators and
policymakers alike came to view information from noncognitive instruments as less credible than the results
from content tests. Tools such as surveys, self-reports,
interviews, and third-party ratings were seen as yielding
information that was distinct and separate from (and
less valuable than) the cognitive data collected by standardized tests.
In essence, researchers and educators created a self-

as cognitive content knowledge represents a subset of
cognition. My definition of metacognitive in this context
includes all learning processes and behaviors involving
any degree of reflection, learning-strategy selection, and
intentional mental processing that can result in a student’s improved ability to learn.
If we were to apply the term metacognitive learning
skills to describe the full range of behaviors, attitudes,
and beliefs students demonstrate while engaging in the
learning process, we could establish semantic parity between cognitive knowledge and noncognitive skills. This
would be a monumental accomplishment that could lead
to a dramatic increase in the development and use of
new tools and techniques designed specifically to help
us develop insight into student learning strategies. Gaining this type of insight would enable educators to teach
students how to learn, as well as what to learn. It would

http://www.edweek.org http://www.edweek.org/go/commentary http://www.edweek.org/go/PISAresponse

Education Week - January 23, 2013

Table of Contents for the Digital Edition of Education Week - January 23, 2013