Quality Progress - January 2016 - 13
BACK TO BASICS
BY JOHN G. SUEDBECK
Assess reliability of audit evidence for effective risk management
IN GENERAL, the audit process is similar
assurance specialist, I made some changes
across most industries. From financial
to the document to create the following
obtained from the machine operator may
audits to quality audits, the following key
guide, which will be useful for assessing the
be more reliable with regard to how well
reliability of quality audit evidence.
a particular machine works than evidence
from the engineer who built the machine.
* Auditing is conducted against an
* Auditors assess controls for adequacy
8. Authoritativeness. Evidence
Guide to assessment
Consider the supplier's evaluation history.
First, consider six categories of evidence
What authority performed the audit or
from which a quality auditor can choose:
* Audits collect evidence.
9. Directness. Interviewing and ob-
* The information obtained is used to
serving the operator perform the task may
assess risk and plan for risk mitigation.
3. Analytical evidence.
be more reliable than reviewing the work
ISO 9001:2008 and ICH Q9 provide guid-
4. Inquiries of the supplier.
order steps. Also, an original document is
ance for the overall process of evaluating a
more reliable than its copy.
supplier. ISO 9001 is a generic, internation-
10. Adequacy of controls. Evidence
from a system or process adequately
ally accepted quality management system
To assess the reliability of the evidence
standard that is relevant to most business-
obtained, we must consider the relevance,
controlled is more reliable than evidence
es. The ICH Q9 guide, "Quality Risk Man-
sufficiency and competence of the evi-
from a poorly controlled or questionable
agement," provides guidance for managing
dence collected. The following guidelines
system or process.
risk based on the same risk management
can help define these attributes.
principles that are effectively used in many
areas of business and government.
The ICH Q9 guide states: "Risk evaluation compares the identified and analyzed
risk against given risk criteria. Risk evalua-
1. Objectivity. Is the evidence objec-
useful for any organization in its quality
achieved when two or more independent au-
audit processes and can benefit its overall
ditors are likely to arrive at the same result.
risk management strategy. Only with reli-
2. Documentation. Documented
evidence, such as records, provides proof
all three of the fundamental questions."2
of compliance to procedures and is more
These fundamental questions are:
reliable than verbal evidence.
2. What is the likelihood (probability) it will
3. What are the consequences (severity)?
To effectively evaluate risk, we need an
understanding of the reliability of the audit
evidence obtained. But how do we best assess the reliability of audit evidence?
After reviewing quality auditing texts
3. Externality. Third-party evidence
may be more reliable than evidence from
within the organization being audited.
4. Sample size. Larger samples may
be more reliable than smaller samples.
5. Sampling method. Was it appropriate?
6. Corroboration. Corroborated evidence is the same or similar to evidence
and articles, I found few resources that
from two or more independent sources. It
answer this question. The most relevant
may be more reliable than uncorroborated
information came from financial auditing
best practices, specifically one article that
financial industry, these guidelines can be
tive or subjective? Objective evidence is
tions consider the strength of evidence for
1. What might go wrong?
Adapted from best practices from the
7. Timeliness. Timely evidence is
provided a guide for assessing the reliability
typically more reliable than evidence
of financial audit evidence.3 As a quality
produced after a delay.
able audit evidence can it assess risk and
mitigate it effectively. QP
REFERENCES AND NOTE
1. ICH Q9 is a Federal Drug Administration standard on
quality risk management developed by the International
Conference on Harmonization of Technical Requirements
for Registration of Pharmaceuticals for Human Use.
Guidances/ucm128053.pdf (case sensitive) for details.
2. "ICH Harmonized Tripartite Guideline: Quality Risk
Management Q9," International Conference on
Harmonization of Technical Requirements for Registration for
Pharmaceuticals for Human Use, November 2005, www.ich.
Quality/Q9/Step4/Q9_Guideline.pdf (case sensitive).
3. Richard L. Ratliff and I. Richard Johnson, "Evidence - Audit
Evidence - Includes Guidance on Audit Evidence," Internal
Auditor, August 1998.
JOHN G. SUEDBECK is a quality
assurance specialist for Metrics Inc.
in Greenville, NC. He earned a bachelor's degree in analytical chemistry
from Fayetteville State University in
North Carolina. A senior member of
ASQ, Suedbeck is an ASQ-certified
quality manager, quality improvement
associate and quality auditor.
January 2016 * QP 13
Table of Contents for the Digital Edition of Quality Progress - January 2016
According to Plan
Use Your Head
Stakeholder Management 101
All About Data
Eight Simple Steps
Which Six Sigma Metric Should I Use?
Turning ‘Who’ Into ‘How’
In the Beginning
Outputs and Outcomes
That’s So Random—Or Is It?
Improving a System
Putting It All on the Table
Know the Drill
It’s Fun To Work With an F-M-E-A
Solve Problems With Open Communication
Tell Me About It
Separate the Vital Few From the Trivial Many
To DMAIC or Not to DMAIC?
Breaking It Down
1 + 1 = Zero Defects
Curve Your Enthusiasm
Make a Choice
What Is a Fault Tree Analysis?
Successful Relationship Diagrams
The Benefits of PDCA
Return on Investment
The Art of Root Cause Analysis
Why Ask Why?
Get to the Root of It
Checks and Balances
Clearing SPC Hurdles
Supplier Selection and Maintenance
Building a Quality Team
Plan Experiments to Prevent Problems
Quality Progress - January 2016