Training Industry Magazine - Winter 2022 - 26

was fun? We can't expect businessminded
bottom-liners to care about a
one-time training experience that is fun,
without telling them about the longerterm
story of impact.
The Value in Level 1
But that doesn't mean you should
scrap your Level 1! All is not lost. It's
still a valuable data point to collect.
The bad news is you need more - but
the very good news is that, despite
assertions to the contrary, there is
actually a correlation between how
engaged the learner is during training
and how much it might later impact that
employee's application (Level 3) and
business benefits (Level 4). The even
better news is that there are ways to
increase the power of your Level 1 by
including all the right questions in the
overall score.
Can Level 1 Reaction Predict
Level 3 Employee Behavior
Change on the Job?
Based on my own research and data
collected over the past 15 years, the
answer is yes, it does have some
predictive power. The reason why it's
been traditionally so frustrating to draw
this conclusion or find research in the
industry is that most Level 1 assessments
don't ask the right questions, nor do
they use the right scaling. Further,
there is rarely any consistency across
companies, or even within companies,
as to how they measure their Level 1
results (our independent variable). Add
to that the scarcity of L&D organizations
that are validly measuring Level 3 (our
outcome variable), and you've got an
almost impossible hypothesis to prove.
So, what evidence do I have of a
correlation? The strength of my findings
comes from the consistency of how
I've collected Level 1-6 data over the
years. That is, I've gathered data and
reported findings on over 150 different
training programs in over a dozen
different companies using the same
methodology, and at least a core set
of assessment questions to capture
Levels 1 and 3. Using this data, the
evidence of a relationship breaks up
into two key findings:
Correlations Within
Programs
Here, the Level 1 scores of thousands
of internal employees (on a five-point
scale) were correlated to their respective
scores for Level 3 (standardized to a
five-point scale) for those very same
training programs. The correlation for
all participants in the study (N=20,116)
was a 0.31, which was weak, but it was
statistically significant considering the
size of the sample.
Differences Across
Delivery Modes
Here, the overall Level 1 scores
for different delivery modes
(our independent variable) were
collected and then compared to the
overall Level 3 scores for those same
delivery modes (dependent variable).
I used an analysis of variance within
and between groups to make sure any
differences were significant and so far,
the evidence is compelling. The main
modes of delivery in the study included
training guides, training videos, selfdirected
web-based training, classroom
training and virtual reality (VR) training.
Essentially, the overall Level 1 scores
went up as the training became more
sophisticated. Simple instructional
training manuals and videos scored the
lowest in " exciting and engaging, " and
VR scored the highest. The next step was
to look at any available Level 3 scores
across all the delivery modes and see if
this progressively higher engagement
translated into a higher impact on the
job. Since there were not many full Level
1-6 studies done for the low-investment
training manuals and videos, I ended up
comparing the aggregate results of just
three different delivery modes - selfSelf-directed
Online
Classroom
Virtual Reality
| 26
3.8
4.4
4.9
3.0
3.9
4.5
Training Industry Magazine - Winter 2022

Table of Contents for the Digital Edition of Training Industry Magazine - Winter 2022
