Training Industry Quarterly - Spring 2008 - (Page 17)

Table 1: The Kirkpatrick/Phillips Five Levels of Evaluation, Plus the crucial sixth level developed by Dr. Paul Leone, American Express The Results For Level 1 and 2, the leadership training received similar positive satisfaction (L1) and acquired knowledge (L2) scores (88% to 90%), regardless of how it was delivered. For Level 3, there was a significant difference based on how it was delivered with the blended clearly resulting in the greatest behavior change. This was measured according to the percentage of leaders showing “high improvement” in specific leadership competencies three months post training (32% online; 42% classroom; 51% blended). For Level 4, we measured the productivity improvements of all of the participants’ direct reports (leadership training should result in a participant’s direct reports being more engaged, motivated and productive). We used a Level 4 calculation to isolate how much of their overall increase could be attributable directly to the training. Again, the blended approach had the most impact (results were 5% increases in productivity for direct reports of Web-based participants; 10% for instructor-led; to 12% for the blended approach). For Level 5 (ROI), because the blended solution had twice the impact of the Web-based and was nearly half the cost of the instructorled, the ROI for the blended was 75% to 100% higher than either Web-based or classroom alone. Measurement Level 1: Reaction Level 2: Knowledge Level 3: Behavior Level 4: Impact Level 5: ROI Level 6: Transfer Climate Method of Evaluation Learner responds to survey upon completion of learning event (satisfaction with experience) Learner responds to survey upon completion of learning event (new knowledge and skills acquired) Assessment completed by manager, learner and direct reports three months after learning event (observed improvement in leadership skills) Assessment completed by manager, learner and direct reports three months after learning event (improved productivity of direct reports) Cost vs. Benefit analysis based on increase in sales productivity of direct reports over three months Assesses factors in learners’work environment (climate) that will help or hinder the transfer of learning To measure effectiveness at the various levels of evaluation, employees participating in the leadership program were first given a survey immediately after training. This survey captured satisfaction with the experience and the level of new knowledge and skills acquired (Levels 1 and 2). Three months after the training was completed, participants were invited to take another survey online and were instructed to invite their leaders and direct reports to complete a similar survey to get multirater feedback. This survey captured behavior change and application back on the job (Level 3), as well as increases in productivity and performance (Level 4). A return on investment (ROI) analysis (Level 5) was then conducted for all participants reporting “sales revenue” as their primary performance measure. Included in the self-assessment across the organization was a “transfer climate index” that asked participants about the work environment they returned to posttraining (Level 6). These Level 1 to 5 findings were extremely valuable, but we didn’t want to stop here. We still wanted to understand what it was about the blended approach that really drove these great results. This is where our Level 6 analysis really paid off. The Level 6 — Transfer Climate measure tells us the extent to which factors in a participant’s immediate work environment were either helping or hindering the “transfer” of the learning back to the job. This additional level of analysis not only lets the organization know why training is far more effective for some groups over others, but also informs business groups of what they can do to maximize the effects of their employees’ training in the future. In this case, the new measure indicated that the blended learning approach was more effective because it created an incredible post-training climate for transferring the learning and applying it back on the job. That is, the majority of participants going through the blended experience had significantly higher support and endorsement from their immediate managers, more frequent one-on-one follow-up conversations with their immediate managers, and had a much higher expectation that they would be Table 2. This illustration depicts just how much these factors (all directly controlled by the immediate manager) could discriminate between levels of impact. For example, 72% of learners in the “high improvement” group were having one-on-one conversations about how to apply, while only 17% of the “no improvement”group were having these conversations. This table shows how all three factors were present and pervasive for the “high improvement” group, while they were all significantly lower for the “no improvement” group. Training Industry Quarterly, Spring 2008 / A Training Industry, Inc. ezine / www.trainingindustry.com/TIQ 17 http://www.trainingindustry.com/TIQ

Table of Contents Feed for the Digital Edition of Training Industry Quarterly - Spring 2008

Training Industry Quarterly - Spring 2008 At the Editor’s Desk Contents Winning Organizations Through People Before You Buy… Learning Technologies Take Your ROI to Level 6 LCMS: A Critical Link to Learning Success Establishing Best Practices for Learning Governance Training’s Role in Continuous Improvement Meet Jim Mitnick Meet Karen Kocher Meet Christina Cernuch The Personal Side of Personnel Training Closing Arguments

Training Industry Quarterly - Spring 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.