Quality Progress - January 2016 - 13

BACK TO BASICS Best of JUNE 2012 BY JOHN G. SUEDBECK Solid Proof Assess reliability of audit evidence for effective risk management IN GENERAL, the audit process is similar assurance specialist, I made some changes across most industries. From financial to the document to create the following obtained from the machine operator may audits to quality audits, the following key guide, which will be useful for assessing the be more reliable with regard to how well principles apply: reliability of quality audit evidence. a particular machine works than evidence from the engineer who built the machine. * Auditing is conducted against an agreed-upon standard. * Auditors assess controls for adequacy and compliance. 8. Authoritativeness. Evidence Guide to assessment Consider the supplier's evaluation history. First, consider six categories of evidence What authority performed the audit or from which a quality auditor can choose: certification? * Audits collect evidence. 1. Confirmations. 9. Directness. Interviewing and ob- * The information obtained is used to 2. Documentation. serving the operator perform the task may assess risk and plan for risk mitigation. 3. Analytical evidence. be more reliable than reviewing the work ISO 9001:2008 and ICH Q9 provide guid- 4. Inquiries of the supplier. order steps. Also, an original document is ance for the overall process of evaluating a 5. Retest. more reliable than its copy. supplier. ISO 9001 is a generic, internation- 6. Observations. 1 10. Adequacy of controls. Evidence from a system or process adequately ally accepted quality management system To assess the reliability of the evidence standard that is relevant to most business- obtained, we must consider the relevance, controlled is more reliable than evidence es. The ICH Q9 guide, "Quality Risk Man- sufficiency and competence of the evi- from a poorly controlled or questionable agement," provides guidance for managing dence collected. The following guidelines system or process. risk based on the same risk management can help define these attributes. principles that are effectively used in many areas of business and government. The ICH Q9 guide states: "Risk evaluation compares the identified and analyzed risk against given risk criteria. Risk evalua- 1. Objectivity. Is the evidence objec- useful for any organization in its quality achieved when two or more independent au- audit processes and can benefit its overall ditors are likely to arrive at the same result. risk management strategy. Only with reli- 2. Documentation. Documented evidence, such as records, provides proof all three of the fundamental questions."2 of compliance to procedures and is more These fundamental questions are: reliable than verbal evidence. 2. What is the likelihood (probability) it will go wrong? 3. What are the consequences (severity)? To effectively evaluate risk, we need an understanding of the reliability of the audit evidence obtained. But how do we best assess the reliability of audit evidence? After reviewing quality auditing texts 3. Externality. Third-party evidence may be more reliable than evidence from within the organization being audited. 4. Sample size. Larger samples may be more reliable than smaller samples. 5. Sampling method. Was it appropriate? 6. Corroboration. Corroborated evidence is the same or similar to evidence and articles, I found few resources that from two or more independent sources. It answer this question. The most relevant may be more reliable than uncorroborated information came from financial auditing evidence. best practices, specifically one article that financial industry, these guidelines can be tive or subjective? Objective evidence is tions consider the strength of evidence for 1. What might go wrong? Adapted from best practices from the 7. Timeliness. Timely evidence is provided a guide for assessing the reliability typically more reliable than evidence of financial audit evidence.3 As a quality produced after a delay. able audit evidence can it assess risk and mitigate it effectively. QP REFERENCES AND NOTE 1. ICH Q9 is a Federal Drug Administration standard on quality risk management developed by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Visit www.fda.gov/downloads/RegulatoryInformation/ Guidances/ucm128053.pdf (case sensitive) for details. 2. "ICH Harmonized Tripartite Guideline: Quality Risk Management Q9," International Conference on Harmonization of Technical Requirements for Registration for Pharmaceuticals for Human Use, November 2005, www.ich. org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/ Quality/Q9/Step4/Q9_Guideline.pdf (case sensitive). 3. Richard L. Ratliff and I. Richard Johnson, "Evidence - Audit Evidence - Includes Guidance on Audit Evidence," Internal Auditor, August 1998. JOHN G. SUEDBECK is a quality assurance specialist for Metrics Inc. in Greenville, NC. He earned a bachelor's degree in analytical chemistry from Fayetteville State University in North Carolina. A senior member of ASQ, Suedbeck is an ASQ-certified quality manager, quality improvement associate and quality auditor. January 2016 * QP 13 http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm128053.pdf http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf

Table of Contents for the Digital Edition of Quality Progress - January 2016

According to Plan
Solid Proof
Use Your Head
Stakeholder Management 101
Customer Delight
All About Data
Eight Simple Steps
Minimizing Chaos
Take Note
Pyramid Scheme
Assessing Failure
Which Six Sigma Metric Should I Use?
Turning ‘Who’ Into ‘How’
In the Beginning
Outputs and Outcomes
That’s So Random—Or Is It?
Understanding Variation
Improving a System
Putting It All on the Table
Know the Drill
It’s Fun To Work With an F-M-E-A
Solve Problems With Open Communication
Tell Me About It
Separate the Vital Few From the Trivial Many
To DMAIC or Not to DMAIC?
Breaking It Down
Smart Charting
1 + 1 = Zero Defects
QFD Explained
Curve Your Enthusiasm
Make a Choice
What Is a Fault Tree Analysis?
Successful Relationship Diagrams
The Benefits of PDCA
Creative Combination
Return on Investment
The Art of Root Cause Analysis
Why Ask Why?
Get to the Root of It
Checks and Balances
Calculated Risk
Sufficient Evidence
Sample Wise
Clearing SPC Hurdles
Supplier Selection and Maintenance
Team Advantage
Building a Quality Team
Plan Experiments to Prevent Problems
Substantiation Test
Training Day

Quality Progress - January 2016

http://www.nxtbook.com/naylor/ASQM/ASQM0719
http://www.nxtbook.com/naylor/ASQM/ASQM0619
http://www.nxtbook.com/naylor/ASQM/ASQM0519
http://www.nxtbook.com/naylor/ASQM/ASQM0419
http://www.nxtbook.com/naylor/ASQM/ASQM0319
http://www.nxtbook.com/naylor/ASQM/ASQM0219
http://www.nxtbook.com/naylor/ASQM/ASQM0119
http://www.nxtbook.com/naylor/ASQM/ASQM1218
http://www.nxtbook.com/naylor/ASQM/ASQM1118
http://www.nxtbook.com/naylor/ASQM/ASQM1018
http://www.nxtbook.com/naylor/ASQM/ASQM0918
http://www.nxtbook.com/naylor/ASQM/ASQM0818
http://www.nxtbook.com/naylor/ASQM/ASQM0718
http://www.nxtbook.com/naylor/ASQM/ASQM0618
http://www.nxtbook.com/naylor/ASQM/ASQM0518
http://www.nxtbook.com/naylor/ASQM/ASQM0418
http://www.nxtbook.com/naylor/ASQM/ASQM0318
http://www.nxtbook.com/naylor/ASQM/ASQM0218
http://www.nxtbook.com/naylor/ASQM/ASQM0118
http://www.nxtbook.com/naylor/ASQM/ASQM1217
http://www.nxtbook.com/naylor/ASQM/ASQM1117
http://www.nxtbook.com/naylor/ASQM/ASQM1017
http://www.nxtbook.com/naylor/ASQM/ASQM0917
http://www.nxtbook.com/naylor/ASQM/ASQM0817
http://www.nxtbook.com/naylor/ASQM/ASQM0717
http://www.nxtbook.com/naylor/ASQM/ASQM0617
http://www.nxtbook.com/naylor/ASQM/ASQM0517
http://www.nxtbook.com/naylor/ASQM/ASQM0417
http://www.nxtbook.com/naylor/ASQM/ASQC12518
http://www.nxtbook.com/naylor/ASQM/ASQM0317
http://www.nxtbook.com/naylor/ASQM/ASQM0217
http://www.nxtbook.com/naylor/ASQM/ASQM0117
http://www.nxtbook.com/naylor/ASQM/ASQM1216
http://www.nxtbook.com/naylor/ASQM/ASQM1116
http://www.nxtbook.com/naylor/ASQM/ASQM1016
http://www.nxtbook.com/naylor/ASQM/ASAC0016
http://www.nxtbook.com/naylor/ASQM/ASQM0916
http://www.nxtbook.com/naylor/ASQM/ASQA0016
http://www.nxtbook.com/naylor/ASQM/ASQM0816
http://www.nxtbook.com/naylor/ASQM/ASQM0716
http://www.nxtbook.com/naylor/ASQM/ASQM0616
http://www.nxtbook.com/naylor/ASQM/ASQM0516
http://www.nxtbook.com/naylor/ASQM/ASQM0416
http://www.nxtbook.com/naylor/ASQM/ASQM0316
http://www.nxtbook.com/naylor/ASQM/ASQM0216
http://www.nxtbook.com/naylor/ASQM/ASQM0116
http://www.nxtbook.com/naylor/ASQM/ASQM1215
http://www.nxtbook.com/naylor/ASQM/ASQM1115
http://www.nxtbook.com/naylor/ASQM/ASQM1015
http://www.nxtbook.com/naylor/ASQM/ASQM0915
http://www.nxtbook.com/naylor/ASQM/ASQM0815
http://www.nxtbook.com/naylor/ASQM/ASQM0715
http://www.nxtbook.com/naylor/ASQM/ASQM0615
http://www.nxtbook.com/naylor/ASQM/ASQM0515
http://www.nxtbook.com/naylor/ASQM/ASQM0315
http://www.nxtbook.com/naylor/ASQM/ASQM0215
http://www.nxtbook.com/naylor/ASQM/ASQM0115
http://www.nxtbook.com/naylor/ASQM/ASQM1214
http://www.nxtbook.com/naylor/ASQM/ASQM1114
http://www.nxtbook.com/naylor/ASQM/ASQM1014
http://www.nxtbook.com/naylor/ASQM/ASQM0914
http://www.nxtbook.com/naylor/ASQM/ASQM0814
http://www.nxtbook.com/naylor/ASQM/ASQM0714
http://www.nxtbook.com/naylor/ASQM/ASQM0614
http://www.nxtbook.com/naylor/ASQM/ASQM0514
http://www.nxtbook.com/naylor/ASQM/ASQM0414
http://www.nxtbook.com/naylor/ASQM/ASQM0314
http://www.nxtbook.com/naylor/ASQM/ASQM0214
http://www.nxtbook.com/naylor/ASQM/ASQM0114
http://www.nxtbook.com/naylor/ASQM/ASQM1213
http://www.nxtbook.com/naylor/ASQM/ASQM1113
http://www.nxtbook.com/naylor/ASQM/ASQM1013
http://www.nxtbook.com/naylor/ASQM/ASQM0913
http://www.nxtbook.com/naylor/ASQM/ASQM0813
http://www.nxtbook.com/naylor/ASQM/ASQM0713
http://www.nxtbook.com/naylor/ASQM/ASQM0613
http://www.nxtbook.com/naylor/ASQM/ASQM0513
http://www.nxtbook.com/naylor/ASQM/ASQM0413
http://www.nxtbook.com/naylor/ASQM/ASQM0313
http://www.nxtbook.com/nxtbooks/naylor/ASQM0213
http://www.nxtbook.com/nxtbooks/naylor/ASQM0113
http://www.nxtbook.com/nxtbooks/naylor/ASQM1212
http://www.nxtbook.com/nxtbooks/naylor/ASQM1112
http://www.nxtbook.com/nxtbooks/naylor/ASQM1012
http://www.nxtbook.com/nxtbooks/naylor/ASQM0912
http://www.nxtbookMEDIA.com