Quality Progress - January 2016 - 13

BACK TO BASICS Best of JUNE 2012 BY JOHN G. SUEDBECK Solid Proof Assess reliability of audit evidence for effective risk management IN GENERAL, the audit process is similar assurance specialist, I made some changes across most industries. From financial to the document to create the following obtained from the machine operator may audits to quality audits, the following key guide, which will be useful for assessing the be more reliable with regard to how well principles apply: reliability of quality audit evidence. a particular machine works than evidence from the engineer who built the machine. * Auditing is conducted against an agreed-upon standard. * Auditors assess controls for adequacy and compliance. 8. Authoritativeness. Evidence Guide to assessment Consider the supplier's evaluation history. First, consider six categories of evidence What authority performed the audit or from which a quality auditor can choose: certification? * Audits collect evidence. 1. Confirmations. 9. Directness. Interviewing and ob- * The information obtained is used to 2. Documentation. serving the operator perform the task may assess risk and plan for risk mitigation. 3. Analytical evidence. be more reliable than reviewing the work ISO 9001:2008 and ICH Q9 provide guid- 4. Inquiries of the supplier. order steps. Also, an original document is ance for the overall process of evaluating a 5. Retest. more reliable than its copy. supplier. ISO 9001 is a generic, internation- 6. Observations. 1 10. Adequacy of controls. Evidence from a system or process adequately ally accepted quality management system To assess the reliability of the evidence standard that is relevant to most business- obtained, we must consider the relevance, controlled is more reliable than evidence es. The ICH Q9 guide, "Quality Risk Man- sufficiency and competence of the evi- from a poorly controlled or questionable agement," provides guidance for managing dence collected. The following guidelines system or process. risk based on the same risk management can help define these attributes. principles that are effectively used in many areas of business and government. The ICH Q9 guide states: "Risk evaluation compares the identified and analyzed risk against given risk criteria. Risk evalua- 1. Objectivity. Is the evidence objec- useful for any organization in its quality achieved when two or more independent au- audit processes and can benefit its overall ditors are likely to arrive at the same result. risk management strategy. Only with reli- 2. Documentation. Documented evidence, such as records, provides proof all three of the fundamental questions."2 of compliance to procedures and is more These fundamental questions are: reliable than verbal evidence. 2. What is the likelihood (probability) it will go wrong? 3. What are the consequences (severity)? To effectively evaluate risk, we need an understanding of the reliability of the audit evidence obtained. But how do we best assess the reliability of audit evidence? After reviewing quality auditing texts 3. Externality. Third-party evidence may be more reliable than evidence from within the organization being audited. 4. Sample size. Larger samples may be more reliable than smaller samples. 5. Sampling method. Was it appropriate? 6. Corroboration. Corroborated evidence is the same or similar to evidence and articles, I found few resources that from two or more independent sources. It answer this question. The most relevant may be more reliable than uncorroborated information came from financial auditing evidence. best practices, specifically one article that financial industry, these guidelines can be tive or subjective? Objective evidence is tions consider the strength of evidence for 1. What might go wrong? Adapted from best practices from the 7. Timeliness. Timely evidence is provided a guide for assessing the reliability typically more reliable than evidence of financial audit evidence.3 As a quality produced after a delay. able audit evidence can it assess risk and mitigate it effectively. QP REFERENCES AND NOTE 1. ICH Q9 is a Federal Drug Administration standard on quality risk management developed by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Visit www.fda.gov/downloads/RegulatoryInformation/ Guidances/ucm128053.pdf (case sensitive) for details. 2. "ICH Harmonized Tripartite Guideline: Quality Risk Management Q9," International Conference on Harmonization of Technical Requirements for Registration for Pharmaceuticals for Human Use, November 2005, www.ich. org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/ Quality/Q9/Step4/Q9_Guideline.pdf (case sensitive). 3. Richard L. Ratliff and I. Richard Johnson, "Evidence - Audit Evidence - Includes Guidance on Audit Evidence," Internal Auditor, August 1998. JOHN G. SUEDBECK is a quality assurance specialist for Metrics Inc. in Greenville, NC. He earned a bachelor's degree in analytical chemistry from Fayetteville State University in North Carolina. A senior member of ASQ, Suedbeck is an ASQ-certified quality manager, quality improvement associate and quality auditor. January 2016 * QP 13 http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm128053.pdf http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf

Table of Contents for the Digital Edition of Quality Progress - January 2016

According to Plan
Solid Proof
Use Your Head
Stakeholder Management 101
Customer Delight
All About Data
Eight Simple Steps
Minimizing Chaos
Take Note
Pyramid Scheme
Assessing Failure
Which Six Sigma Metric Should I Use?
Turning ‘Who’ Into ‘How’
In the Beginning
Outputs and Outcomes
That’s So Random—Or Is It?
Understanding Variation
Improving a System
Putting It All on the Table
Know the Drill
It’s Fun To Work With an F-M-E-A
Solve Problems With Open Communication
Tell Me About It
Separate the Vital Few From the Trivial Many
To DMAIC or Not to DMAIC?
Breaking It Down
Smart Charting
1 + 1 = Zero Defects
QFD Explained
Curve Your Enthusiasm
Make a Choice
What Is a Fault Tree Analysis?
Successful Relationship Diagrams
The Benefits of PDCA
Creative Combination
Return on Investment
The Art of Root Cause Analysis
Why Ask Why?
Get to the Root of It
Checks and Balances
Calculated Risk
Sufficient Evidence
Sample Wise
Clearing SPC Hurdles
Supplier Selection and Maintenance
Team Advantage
Building a Quality Team
Plan Experiments to Prevent Problems
Substantiation Test
Training Day

Quality Progress - January 2016

https://www.nxtbook.com/naylor/ASQM/ASQM0719
https://www.nxtbook.com/naylor/ASQM/ASQM0619
https://www.nxtbook.com/naylor/ASQM/ASQM0519
https://www.nxtbook.com/naylor/ASQM/ASQM0419
https://www.nxtbook.com/naylor/ASQM/ASQM0319
https://www.nxtbook.com/naylor/ASQM/ASQM0219
https://www.nxtbook.com/naylor/ASQM/ASQM0119
https://www.nxtbook.com/naylor/ASQM/ASQM1218
https://www.nxtbook.com/naylor/ASQM/ASQM1118
https://www.nxtbook.com/naylor/ASQM/ASQM1018
https://www.nxtbook.com/naylor/ASQM/ASQM0918
https://www.nxtbook.com/naylor/ASQM/ASQM0818
https://www.nxtbook.com/naylor/ASQM/ASQM0718
https://www.nxtbook.com/naylor/ASQM/ASQM0618
https://www.nxtbook.com/naylor/ASQM/ASQM0518
https://www.nxtbook.com/naylor/ASQM/ASQM0418
https://www.nxtbook.com/naylor/ASQM/ASQM0318
https://www.nxtbook.com/naylor/ASQM/ASQM0218
https://www.nxtbook.com/naylor/ASQM/ASQM0118
https://www.nxtbook.com/naylor/ASQM/ASQM1217
https://www.nxtbook.com/naylor/ASQM/ASQM1117
https://www.nxtbook.com/naylor/ASQM/ASQM1017
https://www.nxtbook.com/naylor/ASQM/ASQM0917
https://www.nxtbook.com/naylor/ASQM/ASQM0817
https://www.nxtbook.com/naylor/ASQM/ASQM0717
https://www.nxtbook.com/naylor/ASQM/ASQM0617
https://www.nxtbook.com/naylor/ASQM/ASQM0517
https://www.nxtbook.com/naylor/ASQM/ASQM0417
https://www.nxtbook.com/naylor/ASQM/ASQC12518
https://www.nxtbook.com/naylor/ASQM/ASQM0317
https://www.nxtbook.com/naylor/ASQM/ASQM0217
https://www.nxtbook.com/naylor/ASQM/ASQM0117
https://www.nxtbook.com/naylor/ASQM/ASQM1216
https://www.nxtbook.com/naylor/ASQM/ASQM1116
https://www.nxtbook.com/naylor/ASQM/ASQM1016
https://www.nxtbook.com/naylor/ASQM/ASAC0016
https://www.nxtbook.com/naylor/ASQM/ASQM0916
https://www.nxtbook.com/naylor/ASQM/ASQA0016
https://www.nxtbook.com/naylor/ASQM/ASQM0816
https://www.nxtbook.com/naylor/ASQM/ASQM0716
https://www.nxtbook.com/naylor/ASQM/ASQM0616
https://www.nxtbook.com/naylor/ASQM/ASQM0516
https://www.nxtbook.com/naylor/ASQM/ASQM0416
https://www.nxtbook.com/naylor/ASQM/ASQM0316
https://www.nxtbook.com/naylor/ASQM/ASQM0216
https://www.nxtbook.com/naylor/ASQM/ASQM0116
https://www.nxtbook.com/naylor/ASQM/ASQM1215
https://www.nxtbook.com/naylor/ASQM/ASQM1115
https://www.nxtbook.com/naylor/ASQM/ASQM1015
https://www.nxtbook.com/naylor/ASQM/ASQM0915
https://www.nxtbook.com/naylor/ASQM/ASQM0815
https://www.nxtbook.com/naylor/ASQM/ASQM0715
https://www.nxtbook.com/naylor/ASQM/ASQM0615
https://www.nxtbook.com/naylor/ASQM/ASQM0515
https://www.nxtbook.com/naylor/ASQM/ASQM0315
https://www.nxtbook.com/naylor/ASQM/ASQM0215
https://www.nxtbook.com/naylor/ASQM/ASQM0115
https://www.nxtbook.com/naylor/ASQM/ASQM1214
https://www.nxtbook.com/naylor/ASQM/ASQM1114
https://www.nxtbook.com/naylor/ASQM/ASQM1014
https://www.nxtbook.com/naylor/ASQM/ASQM0914
https://www.nxtbook.com/naylor/ASQM/ASQM0814
https://www.nxtbook.com/naylor/ASQM/ASQM0714
https://www.nxtbook.com/naylor/ASQM/ASQM0614
https://www.nxtbook.com/naylor/ASQM/ASQM0514
https://www.nxtbook.com/naylor/ASQM/ASQM0414
https://www.nxtbook.com/naylor/ASQM/ASQM0314
https://www.nxtbook.com/naylor/ASQM/ASQM0214
https://www.nxtbook.com/naylor/ASQM/ASQM0114
https://www.nxtbook.com/naylor/ASQM/ASQM1213
https://www.nxtbook.com/naylor/ASQM/ASQM1113
https://www.nxtbook.com/naylor/ASQM/ASQM1013
https://www.nxtbook.com/naylor/ASQM/ASQM0913
https://www.nxtbook.com/naylor/ASQM/ASQM0813
https://www.nxtbook.com/naylor/ASQM/ASQM0713
https://www.nxtbook.com/naylor/ASQM/ASQM0613
https://www.nxtbook.com/naylor/ASQM/ASQM0513
https://www.nxtbook.com/naylor/ASQM/ASQM0413
https://www.nxtbook.com/naylor/ASQM/ASQM0313
https://www.nxtbook.com/nxtbooks/naylor/ASQM0213
https://www.nxtbook.com/nxtbooks/naylor/ASQM0113
https://www.nxtbook.com/nxtbooks/naylor/ASQM1212
https://www.nxtbook.com/nxtbooks/naylor/ASQM1112
https://www.nxtbook.com/nxtbooks/naylor/ASQM1012
https://www.nxtbook.com/nxtbooks/naylor/ASQM0912
https://www.nxtbookmedia.com