Education Week - October 2, 2019 - 14

How Schools Evaluate Impact of Digital Learning Tools
CONTINUED FROM PAGE 1

bers for software, a problem driven
largely by districts not thinking
through how an ed-tech product
should be used prior to buying it and
then setting realistic expectations for
usage, experts say.
Complicating matters is the fact that
there isn't a clear consensus on just
what constitutes a "good" usage rate:
Is it 25 percent, 50 percent, or 100 percent? And experts and educators are
divided on whether education technology leaders should even be worried
about low usage rates.
To solve this problem, experts say
the best way to get the best bang for
your education technology buck is to
set expectations for how often a particular piece of software should be used.
Then keep careful track of how close
you come to your goal. If a program
isn't coming close to meeting that goal,
it's probably time to jettison it.
But that's easier said than done.
There's no silver bullet when it
comes to evaluating which software
to keep and which to ditch, said Joshua
Patchak, the executive director of education technology and innovation for
the Green Bay Area public school district in Wisconsin.
"If it is magic, I would love for someone to teach me those spells," Patchak
said. "I don't know the secret to it. It's
difficult, unless you have a very disciplined staff from the classroom all the
way up to the top."

'Waste of Resources'
Most software licenses districts buy
never get used, according to a November 2018 report by BrightBytes,
an education data-management and
-analytics platform. In fact, the study
found a median of 30 percent of edtech licenses are never used at all.
What's more, nearly all-98 percent-of licenses aren't used intensively, meaning a student spends
10 hours or more with the product
between assessments. And most
teachers aren't following vendors'
recommendations when it comes to
"dosage"-how often and for how
much time a student uses a particular
program or product.
Those findings are based on a set of
data from 48 school districts, 393,000
students, and 177 apps-which BrightBytes defines as browser-based online
curricular and learning tools. The districts studied were of different sizes,
with enrollments from more than
30,000 students to fewer than 1,000.
Similarly, a study released earlier
this year by the ed-tech company
Glimpse K12 that looked at $2 billion
in school spending found that, on average, 67 percent of educational software product licenses go unused.
Glimpse K12 tracked 200,000 curriculum-software licenses purchased
by 275 schools during the 2017-18
school year. The analysis found educational software was the biggest source
of wasted spending in K-12 districts.
And it estimated that, overall, districts
are losing a whopping $2 million each

Apps With Highest Number of 'Intensive' Users in K-12
"Intensive" use means a student spends 10 hours or more using the product between
assessments, according to a 2018 study.

Intensive users
Users

303,357

ers who might not be quick to adapt to
technological advances and some who
might resist using them altogether.
"We have a much older workforce
than a lot of other organizations and
industries," said Matthew Lentz, the
chief financial officer for the Upper
Moreland district, near Philadelphia.
That means some educators aren't as
"nimble" on "how you can adapt in a
changing environment."

What Products to Jettison

154,446

40,783

46,827

40,783
23,244

Google Drive

Canvas

Dreambox Learning

55,432
16,059

24,680

Lexia Reading Core5

15,011

IXL

SOURCE: BrightBytes, "The 2018 Technology & Learning Insights Report:
Towards Understanding App Effectiveness and Cost"

5 Strategies
For Ed-Tech Use

Districts spend millions
on educational technology
products and services. But
many of those products and
services are not used by
teachers or students as much
as you might expect. Here's
some advice from ed-tech
experts and educators on
how to pump up those usage
rates:
n
SOURCE: Education Week

on these products every year.
Those data depress Anton Inglese,
the chief financial officer for Batavia
Public School District 101, in a Chicago
suburb.
"I think it's a waste of resources,"
he said. "We clamor all the time about
how we don't have enough resources
to meet the needs of our children,
and yet if we're investing in resources
and they go unused, I think that's
unconscionable."
But Richard Culatta, the CEO of the
International Society for Technology
in Education, said looking just at how
often a particular product is used is the
wrong way to think about the problem.
"I actually don't like using usage as
a metric," he said. "It implies that we
want everybody to use the software
that we bought. What we should be
thinking about is: Are the right kids and
the right teachers using the right software at the right time? There's rarely a

14 | EDUCATION WEEK | October 2, 2019 | www.edweek.org

1.
2.
3.

Give educators meaningful opportunities to offer
feedback on potential ed-tech purchases.
Listen carefully to educators' critiques of the ed-tech
products and services currently used in the district.

After you buy an ed-tech product or service, establish
a detailed plan for showing all educators why you
purchased it and how to use it. Have a usage plan or goal in
place and continually check in and measure progress against
the goal.

If you expect the software or application to be used
for a certain amount of time, or for a specific purpose,
make sure that expectation matches what's actually going
on in the classroom. For instance, if a teacher is only going
to use a piece of software as an option in a lesson rotation
and some students won't have access to it for weeks, but the
vendor is recommending an hour of use per week for each
student, it might not be the best fit.

Pilot software and structure those pilots in a way
that district leaders and teachers can see what the
student outcomes are likely to be, perhaps by using a control
group and a treatment group. Districts should make sure
that they are testing out a particular piece of software or
service in an environment similar to where it will eventually
be used, since some tools might work well with some types
of students but not so well with others.

case where the answer should be 100
percent, or having more is better."
Instead of asking how often a particular piece of software is being used,
Culatta said, district officials should
ask: "Is it meeting the learning need?"

Reasons for Low Usage
One potential reason for low usage
rates: District leaders in charge of buying software can't always consult with
every teacher to figure out what meets
his or her needs.
In many cases, "school districts have
to make one-size-fits-all decisions,"
said Dan Carroll, the chief product officer of Clever, an education technology
company that has helped track usage
rates in the past. Given that reality, it
"actually feels a little bit inevitable that
you're not going to get 100 percent of
people using your software."
Of course, in his years working with

districts, Carroll, a former tech director, said he's also "seen a lot of dysfunction" where software or an application is purchased sometime over the
summer and not a single teacher uses
it for the entire school year, sometimes
because they don't know it even exists.
But many education technology
leaders will allow teachers or principals
to buy software that not everyone may
be interested in because they don't
want to close the door on experimentation with new applications.
"That's obviously a tricky situation
when the tech evolves so fast. If you
lock the door and don't let anything
in, you miss opportunities," said Hal
Friedlander, the founder and CEO of
the nonprofit Technology for Education Consortium and a former chief
technology officer for the New York
City schools. "It's a tough job."
Another contributing factor that
district officials point to: aging work-

Part of the Batavia district's strategy: classifying curriculum materials
into three categories, including core
materials that get widespread use,
supplemental materials for remediation or enrichment, and materials that
might just meet an individual school,
student, or teacher's needs. The latter
category gets the most scrutiny, with
the goal of shifting materials in it to
one of the other categories.
There's no secret sauce in trying to
decide what to jettison, Inglese, the
district's CFO, said. "There aren't universal criteria you can apply to every
situation. You have to delve into the
particulars of each situation to make
decisions about whether or not these
things work."
And then there's giving school leaders direct responsibility for deciding
which technology will work best for
their schools. In the IDEA charter network in Louisiana and Texas, principals can choose to buy software using
discretionary funds. But that money
doesn't have to go to technology-it
could go to other priorities.
That means if a particular product
isn't being used or isn't effective, principals are going to hear about it.
"They would get that feedback from
their teachers and the staff," said Cody
Grindle, the senior vice president of
information systems at IDEA, which
serves 53,000 students in 97 schools.
"That's a lot easier and [more] successful than me sitting in my ivory tower
saying, 'You're only using 20 percent.'
We do share data around that stuff, but
we're not the ones slapping the wrist."
What's more, the amount of time
a student spends on a particular application may not actually say much
about whether it is effective, said Karl
Rectanus, the co-founder and CEO of
LearnPlatform. For instance, one district that partners with LearnPlatform
found its students were getting much
better results by using a product once
a week, then supplementing with other
materials, as opposed to the five times
a week recommended by the vendor.
Some products are designed to save
time, so you don't necessarily want
students spending huge amounts
of time using them, he said. In his
mind, "good usage is the usage that
matches instruction and gets the best
outcomes for students."
See our online-only special
report about why ed tech
is not used and what to
do about it.

> www.edweek.org/
edtechuse

http://www.edweek.org/edtechuse http://www.edweek.org/edtechuse http://www.edweek.org

Education Week - October 2, 2019

Table of Contents for the Digital Edition of Education Week - October 2, 2019