Education Week - February 10, 2016 - 11

PARCC 2014 - 2015 ONLINE VS. PENCIL-AND-PAPER
TESTS, STATEWIDE RESULTS IN ILLINOIS
50

ELA Scores
43
36

Math Scores

42
37

Paper

36
32

28 29

28
25

31
27 29

30
25

Percent Proficient

Online

Total
Tested

3rd
Grade

4th
Grade

5th
Grade

6th
Grade

7th
Grade

8th
Grade

High
School

Total
Tested

3rd
Grade

4th
Grade

5th
Grade

6th
Grade

7th
Grade

8th
Grade

High
School

SOURCE: Illinois State Board of Education

Paper-Test Takers Get the Edge on Some PARCC Exams
CONTINUED FROM PAGE 1

determine the cause of those score
differences.
M e a n w h i l e, i n M a r y l a n d 's
111,000-student Baltimore County
schools, district officials found similar differences, then used statistical
techniques to isolate the impact of
the test format. They found a strong
"mode effect" in numerous gradesubject combinations: Baltimore
County middle-grades students who
took the paper-based version of the
PARCC English/language arts exam,
for example, scored almost 14 points
higher than students who had equivalent demographic and academic
backgrounds but took the computerbased test.
"The differences are significant
enough that it makes it hard to
make meaningful comparisons between students and [schools] at
some grade levels," said Russell
Brown, the district's chief accountability and performance-management officer. "I think it draws into
question the validity of the first
year's results for PARCC."
Last school year, roughly 5 million
students across 10 states and the
District of Columbia sat for the first
official administration of the PARCC
exams, which are intended to align
with the Common Core State Standards. Nearly 81 percent of those
students took the exams by computer.
Scores on the exams are meant
to be used for federal and state
accountability purposes, to make
instructional decisions at the district and school levels, and, in some
cases, as an eventual graduation
requirement for students and an
evaluation measure for teachers and
principals.
Several states have since dropped
all or part of the PARCC exams,
which are being given again this year.
PARCC officials are still working to determine the full scope and
causes of last year's score discrepancies, which may partly result from
demographic and academic differences between the students who

took the tests on computers and
those who took it on paper, rather
than the testing format itself.
Assessment experts consulted by
Education Week said the remedy for
a "mode effect" is typically to adjust
the scores of all students who took
the exam in a particular format, to
ensure that no student is disadvantaged by the mode of administration.
PARCC officials, however, said they
are not considering such a solution. It
will be up to district and state officials
to determine the scope of any problem
in their schools' test results, as well as
what to do about it, Nellhaus said.
Such uncertainty is bound to create headaches for education leaders, said Michael D. Casserly, the
executive director of the Council of
the Great City Schools, which represents 67 of the country's largest
urban school systems.
"The onus should be on PARCC to
make people aware of what these effects are and what the guidelines are
for state and local school districts to
adjust their data," Casserly said.

Comparing Online and Paper Tests
The challenges associated with
comparing scores across traditional
and technology-based modes of test
administration are not unique to
PARCC.
The Smarter Balanced Assessment Consortium, for example, told
Education Week that it is still investigating possible mode effects in the
results from its 2014-15 tests, taken
by roughly 6 million students in 18
states. That consortium-which,
like PARCC, offers exams aligned
with the common core-has yet to
determine how many students took
the SBAC exam online, although the
proportion is expected to be significantly higher than in PARCC states.
Officials with Smarter Balanced
are in the early stages of preparing
technical reports on that and other
matters.
States that administer their own
standardized exams, meanwhile,
have for years conducted compa-

rability studies while making the
transition from paper- to computerbased tests, generally finding either
a slight advantage for students who
take the tests with paper and pencil, or no statistically significant differences in scores based on mode of
administration.
The National Center for Education Statistics, meanwhile, has
found some signs of mode effects
during its ongoing, multi-year effort
to transition the National Assessment of Educational Progress, or
NAEP, from paper to digital-administration platforms.
There are signs that particular
subgroups of students perform better on a particular testing format
said the acting NCES commissioner,
Peggy G. Carr. And across the board,
students' prior exposure to and experience with computers appears to
impact scores.
"If you are a white male and I am
a black female, and we both have
familiarity with technology, we're
going to do better [on digitally based
assessment items] than our counterparts who don't," she said.
The Illinois state school board
made its PARCC results public in
mid-December.
A comparison of online and paperand-pencil scores done by the state
board's data-analysis division was
also posted on the board's website,
but does not appear to have been
reported on publicly.
That analysis shows often-stark
differences by testing format in the
percentages of Illinois students who
demonstrated proficiency (by scoring
a 4 or 5) on PARCC English/language
arts exams across all tested grades.
Of the 107,067 high school students
who took the test online, for example,
32 percent scored proficient. That's
compared with 50 percent for the
17,726 high school students who took
the paper version of the exam.
The differences by format are not
so pronounced in elementary-grades
math; in grades 3-5, in fact, slightly
higher percentages of students
scored proficient on the online ver-

sion of the PARCC exam than on
the paper version.
But proficiency rates among paper-and-pencil test-takers were 7 to
9 points higher on the 8th grade and
high school math exams.
The Illinois board has not conducted any further analysis of the
results to determine the cause of
those discrepancies. Board officials
declined to be interviewed.
"Both versions of the test provide
reliable and valid information that
teachers and parents can use to
identify student strengths and areas
needing improvement," according to
a statement from the board.

Complicated Transition
In Maryland, meanwhile, more than
41,000 Baltimore County students in
grades 3-8 took the PARCC exams
in 2014-15. Fifty-three percent of
students took the math exam online,
while 29 percent took the English/
language arts exam online. The mode
of test administration was decided on
a school-by-school basis, based on the
ratio of computers to students in each
building's largest grade.
Like Illinois, Baltimore County
found big score differences by mode of
test administration. Among 7th graders, for example, the percentage of
students scoring proficient on the ELA
test was 35 points lower among those
who took the test online than among
those who took the test on paper.
To identify the cause of such discrepancies, district officials compared how students and schools
with similar academic and demographic backgrounds did on each
version of the exams.
They found that after controlling
for student and school characteristics, students were between 3 percent and 9 percent more likely to
score proficient on the paper-andpencil version of the math exam, depending on their grade levels. Students were 11 percent to 14 percent
more likely to score proficient on the
paper version of the ELA exam.
"It will make drawing comparisons

within the first year's results difficult,
and it will make drawing comparisons
between the first- and second-year
[PARCC results] difficult as well," said
Brown, the accountability chief for the
Baltimore County district.
Despite such differences, the test
results still have value, said Nellhaus of PARCC.
"This is still useful and important
information providing a wealth of information for schools to improve instruction and identify students who
need assistance or enrichment," he
said.
But possible mode effects on multistate-consortia exams should be
taken seriously, at least in the short
term, and especially if they have not
been accounted for before test results
are reported publicly, said assessment
experts consulted by Education Week.
"Because we're in a transition stage,
where some kids are still taking
paper-and-pencil tests, and some are
taking them on computer, and there
are still connections to high stakes
and accountability, it's a big deal," said
Derek Briggs, a professor of research
and evaluation methodology at the
University of Colorado at Boulder.
Still, Nellhaus said no consortiumwide score adjustment is currently
under consideration. "This needs to
be handled very locally," he said.
Putting that burden on states and
school districts will likely have significant implications on the ground,
said Casserly of the Council of the
Great City Schools.
"I think it will heighten uncertainty, and maybe even encourage
districts to hold back on how vigorously they apply the results to their
decisionmaking," he said.
Associate Editor Catherine Gewertz
contributed to this article.
Coverage of the implementation of
college- and career-ready standards
and the use of personalized learning
is supported in part by a grant from
the Bill & Melinda Gates Foundation.
Education Week retains sole editorial
control over the content of this coverage.

EDUCATION WEEK | February 10, 2016 | www.edweek.org | 11

http://www.edweek.org

Education Week - February 10, 2016

Table of Contents for the Digital Edition of Education Week - February 10, 2016