Chief Learning Officer - February 2008 - (Page 44)

productivity in practice: The Regence Group: Blended Measurement Chris Moore Created in 1995, The Regence Group is one of the largest affiliations of health care plans in the U.S. Pacific Northwest region, collecting $6.5 billion in annual premiums and serving nearly 3 million members across Oregon, Washington, Idaho and Utah. Customer satisfaction is a top priority at the company, and executives realize high satisfaction levels are directly linked to the readiness of their employees. Ongoing learning initiatives with appropriate and timely end-of-course surveys help the organizational development (OD) team at Regence collect learner feedback and ensure that training is both aligned with the business strategy and effective in transferring the required knowledge and skills. In its surveys, Regence finds it best not to rely solely on Likert scale questions to characterize the success for its training initiatives. Using assessment software, OD adds a series of open-ended questions into its surveys in an effort to solicit anecdotal feedback from its audience. That’s not to say that Likert scale questions don’t have their time and place at Regence. Most students can and will complete them at the end of a course with little effort. However, while Likert-based questions are easier to quantify, inconsistencies can arise because of the minimal thought needed to “shade the box.” In contrast, open-ended questions require more thoughtful, methodical and honest input from employees. Regence employs a clever approach to quantify and automate its review of its electronically captured feedback. “We gather and interpret quantitative and qualitative data for regulatory and ethical compliance, business-process improvement, professional development, training in the right topics and user readiness for enterprisewide systems,” said Jim Baker, senior e-learning specialist. “We gather metrics and narrative responses through the pull-down list, survey matrix, customizable Likert scales, ranking order, multiple-choice and true-false and yes-no, Flash, multiple response, fill-in-the-blank and essay questions. Except for the essay questions, where we already gather anecdotal information, we add a comment box to other types of questions. We share with management the metric fluctuations and trends in employee opinions.” At Regence, anecdotal feedback is analyzed both quantitatively and qualitatively to pinpoint topics that are of particular interest or concern. Quantitatively: • Since people will spend time writing about something they care about, a pattern of long responses to a given question indicates a high level of interest in it. • Large quantities of open-ended responses to a given question help identify areas of interest. • Scanning the number and length of responses can quickly pinpoint the most compelling issues. Qualitatively: • Comments are exported to Microsoft Word where there’s a search for emotive words like “never” and “always.” These words often reflect strong feelings about a subject, e.g., “It was never this way before!” • Strongly worded comments that match up with low ratings on the Likert scale can help prioritize issues for management. • A representative selection of comments is provided at the end of statistical reports. The reasons for low and high scores on a survey are often revealed in the text responses. Tuning in and listening to employee feedback, whether audibly or electronically, is vital to the ongoing success of learning initiatives at Regence. Its keen sense of the words and phrases that imply value and meaning to its employees gives the company insight into continuous improvement of OD’s initiatives, ensuring they provide equal benefits to their clients and their employees. February 2008 anecdotal feedback is. In these cases, your best bet is to compile the feedback and create a story out of it. Then, share that story with your stakeholders to get agreement on what the overall feedback really was and what you’re going to do about it. Real stories with or without metrics can be a very powerful tool for gaining agreement with stakeholders while showing your own focus on your internal customers. Varieties of Anecdotal Evidence Anecdotal feedback also is helpful in other ways. It provides the opportunity to make quick, low-cost corrections that can resolve minor issues with new learning programs. Organizations seem concerned about anonymity when collecting feedback electronically, as they claim it increases the genuineness of responses. Although some organizations capture various demographics from students, some still are reluctant to capture those demographics that may narrow down any ability to isolate who it might be that is responding, regardless of the demographics’ value in assessing and improving impact. The value of anonymous evaluations across the board is debatable. Learning needs accountability if it wants to be taken seriously by the rest of the business. Some of the most valuable feedback comes right off the cuff, face to face, and there’s nothing anonymous about that. Put yourself in front of a group of senior managers or executives for a learning opportunity. You’re almost guaranteed to get unsolicited feedback. Front-line workers or new hires are a little more reserved and not as willing to simply voice their opinion. But when asked, most are more than willing to contribute feedback in some ways to gain recognition, particularly those who really “get it” in regard to the learning content. With anonymous feedback, it becomes difficult to investigate things further when you don’t know who is providing it. So, how do you get the same candid feedback using online survey tools? Surveys based on a five-point Likert scale are good for producing basic aggregation and trending across broad curriculums, demographics and the like. Adding more open-ended questions helps to draw out feedback from the I www.clomedia.com I Chief Learning Officer 44 http://www.clomedia.com

Table of Contents Feed for the Digital Edition of Chief Learning Officer - February 2008

Chief Learning Officer - February 2008 Editor's Letter Table of Contents Business Impact Trends Best Practices Effectiveness Guest Editorial Learning Solutions Home Depot: Building Better Associates CLO Profile Environment Realizing the Vision of “One Philips” Tactics Sun Microsystems’ Next-Generation Worker Video Game Recruiting Tool Productivity The Regence Group: Blended Measurement Human Capital NASA: A Case Study in Technical Leadership Development Case Study Business Intelligence Advertisers' Index Editorial Resources In Conclusion

Chief Learning Officer - February 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.