Training Industry Magazine - July/August 2019 - 48

Approach:
We helped program leaders answer the
second question. Almost no data existed
on diagnostic factors shown by research
to impact learning. We decided on two
strategies - analyzing archival learning
outcome data and collecting survey data
focused on diagnostic factors from future
learners and instructors.
We had information about the training's
objectives,
structure,
stakeholders
and context as well as learners' class
assignments and end-of-course (EOC)
proficiency scores. This allowed us to
determine how much individual and
class-level characteristics impacted
proficiency scores.
The nested structure of learning events
provides the opportunity to explore
sources of influence on outcomes, even in
the absence of direct data on diagnostic
factors associated with a level of analysis
(e.g., class). For our client, each learner
was nested within a class and each class
within an event. Learners and classes were
also nested within instructors, as each
instructor taught multiple classes. 

Results:
Our analyses provided evidence that
instructors contributed strongly to
learners' success in developing proficiency.
For example, instructors accounted for
42% of differences (i.e., variance) in learner
reading proficiency scores.

SO, INSTRUCTORS IMPACT
LEARNING. NOW WHAT?
Identifying instructors as a lever to
improve learning outcomes gave us a
diagnostic factor to focus on ... but now
what? No specific instructor data existed
to guide the creation of diagnostic survey
items or of interventions to improve
instruction. We determined what factor
needed improving, but we still had to
determine how to improve it.

How Do Instructors
Impact Learning?
Instructors impact learning directly
through their decisions and actions

| 48

in preparing and delivering training
content and by interacting with learners.
We defined and measured instructor
performance - the decisions, actions
or behaviors under instructors' control
related to their roles and objectives
in the learning enterprise - not
instructor characteristics.

Will Instructors Differ
on Performance?
Since instructors were content subject
matter experts (SMEs) with varying
degrees of instructional experience, it
was reasonable to assume there would
be performance variability. Without
instructor variability, this approach does
not work.

INSTRUCTORS ARE IMPACT
MULTIPLIERS THROUGH THEIR
INFLUENCE ON LEARNERS
AND NEED INSIGHTS,
TOOLS AND SUPPORT TO
MAXIMIZE THEIR IMPACT.
Defining Instructor Performance:
We reviewed research to identify
instructional behaviors empirically linked
to learner outcomes that could be rated
by learners, instructors and/or supervisors.
We identified behaviors that fit into four
performance domains:

Measuring Instructor
Performance:
We also developed and validated instructor
performance metrics, which assessed
key behaviors in the four domains. Then,
we collected data multiple times during
and at the end of the training for two
complete cycles. The metrics performed as
designed with excellent construct validity
and reliability.

Does Instructor Performance
Impact Learning?
Performance ratings collected throughout
the course, starting at the 25%
course completion mark, significantly
correlated with EOC outcomes. When we
retrospectively compared the performance
ratings of instructors who had high and
low-proficiency classes, instructors who
taught high-proficiency classes had higher
ratings on all items, across all time points;
higher performing instructors had higher
performing learners.
With such robust findings, we developed
and piloted a feedback intervention. We
distributed a feedback report to instructors
with results from the 25% collection,
offered guidance on interpreting its results
and suggested improvement resources.
When we had data from four training
cohorts (two with feedback, two
without), we compared instructors who
received feedback to those who did
not. Instructors who received feedback
improved their subsequent performance
ratings, and their learners had higher
EOC assessment scores.

* Learner Engagement
* Classroom Management
* Responsiveness to Learners
* Adapting to Learner Needs
Over the years, we identified additional
performance domains, but these four
remained relevant for instructor-led
training (ILT). Training context and content,
instructor
effectiveness
measure(s),
instructional philosophy, and learner and
instructor populations all impact what
performance domains are relevant.

Intervention:
We implemented a formative evaluation
and feedback program to deliver results
and provide tools for reflection and
improvement/development planning.
The reports provided comparisons
to help instructors determine if they
needed to improve. Instructors used
the report to guide conversations
about development with supervisors.
Supervisors used the reports to
identify instructors for observation and
coaching. The reports later transitioned
to web-based dashboards.
Training Industry Magazine - July/August 2019

Table of Contents for the Digital Edition of Training Industry Magazine - July/August 2019
