BISA Magazine - Quarter 4, 2017 - 28

COMPLIANCE & REGULATORY WATCH
Federal Reserve guidelines. For a number of reasons, a bank may decide that
the project will not fully conform to
those guidelines; some of the work may
be done by bank staff and other work
by a consulting firm. It is essential that
all work that to be performed, as well
as work that is excluded, be clearly
described at the point of approval and
documented in the final report.
Documentation should also identify
overall responsibility for the project
and specific responsibilities for key
phases, particularly when work is done
by third-party firms. When parts of an
overall Model Validation are done by
separate groups, the overall project
plan should document how the separate reports will be reviewed and relied
upon to issue an opinion.

implemented, date of last validation and other pertinent notes
*

Gather and evaluate the change
control process used for model
development and implementation.

*

Obtain the AML Risk Assessment,
business requirements document
and other information that describes the rationale for the models.

*

Document and perform an assessment of overall governance related
to Model Risk Management, policies and controls.

*

Before analyzing models, the model development process must be
reviewed to assure that it meets the
guidelines. For models provided by
third parties, the bank must receive
sufficient information to judge conformance. Some of the information
that should be reviewed includes:
Biographies of key individuals
involved with model development, documentation of all models
describing their purpose, expected
results and control parameters or
other mechanisms that influence
processing results.

The project plan should be prepared
prior to commencement. It should detail
all major tasks, responsibilities and timelines for completion. Periodic review
of progress against the plan should be
undertaken and documented.
The validation should be guided by a
comprehensive framework, which should
be applicable to any independent assessment and also contain the detailed
requirements for a Model Validation. The
framework should address: key stakeholders and their involvement; project
planning and reporting; data confidentiality and security protocols; required tools
and other supports, information gathering process; methods of analysis; list of
all documentation and artifacts needed;
test period, plans, cases, and results;
report preparation, review, and approval;
work paper management; and methodology for following up on findings.
Review
It is necessary to have your own review
process in place to ensure accuracy.
*

Develop or update the catalog of
all models used by the AML compliance group. The catalog should
contain: Name of the model, purpose, description of its operation,
data requirements, control parameters, expected results, scheduling,
date validated prior to use, date

28
www.bisanet.org

*

Obtain and review all prior Model
Validation and User Acceptance
Test reports.

*

For all models, review and assess
the soundness of implementation.

Testing
All tests performed of models should
be conducted with a data repository
developed specifically for Model Validation. The data repository should be
constructed to support all tests and the
range of outcomes expected.
A statistically valid methodology for data
sampling must be determined and documented. Data sampling may be needed
for various test goals, so sampling
methodology should be consistent with
the data, volumes and test goals.
The data quality review should address
the accuracy of data elements used by
the models and completeness of the
data. Often, the details from the originating message are separate from the

recording of the banking transaction. In
this case, are the two sources combined
appropriately? Also consider that data
elements may be optional in the originating system. If so, is this addressed in
the AML system?
Consistency is also important. You need
to ensure that all the same data values
are represented in a consistent manner.
Tests should be conducted to verify that
all models perform in accordance with
their intended functionality. This can
be accomplished by "back testing." One
method is to use a tool that simulates
how each model should perform, and
then apply the simulation to historical
data to assure the same results that
were originally obtained against historical data are again received. Explain all
variations as acceptable or not.
In addition to back testing, a series of
"Above the Line" and "Below the Line"
tests should be performed. These tests
evaluate each model's behavior against
changes in their thresholds or other
control mechanisms; they also assess
the threshold changes on false positives
and false negatives. One way of visualizing the impact of the changes is with
a Disposition Curve as it will show the
increasing or decreasing rate of productive alerts for the threshold changes.
Analysis
The analysis process should be documented and comprehensive. The specific analytical methods will vary based
on the factors discussed above but
should include the assessment, review
and explanation of: differential alerts
from Above and Below the Line Testing; change in the number of productive alerts as thresholds are modified;
productivity levels against AML risk
assessment and acceptable risk; disposition curves showing the relationship
between parameter changes and productive alerts; false positives and false
negatives; time spent on various alert
types; and review of quality controls.
For information about ACA Compliance
Group or ACA Telavance, contact Dee
Stafford at dstafford@acacompliancegroup.com or 561-628-5288. â–˛


http://www.bisanet.org

Table of Contents for the Digital Edition of BISA Magazine - Quarter 4, 2017

Table of Contents
BISA Magazine - Quarter 4, 2017 - Cover1
BISA Magazine - Quarter 4, 2017 - Cover2
BISA Magazine - Quarter 4, 2017 - Table of Contents
BISA Magazine - Quarter 4, 2017 - 2
BISA Magazine - Quarter 4, 2017 - 3
BISA Magazine - Quarter 4, 2017 - 4
BISA Magazine - Quarter 4, 2017 - 5
BISA Magazine - Quarter 4, 2017 - 6
BISA Magazine - Quarter 4, 2017 - 7
BISA Magazine - Quarter 4, 2017 - 8
BISA Magazine - Quarter 4, 2017 - 9
BISA Magazine - Quarter 4, 2017 - 10
BISA Magazine - Quarter 4, 2017 - 11
BISA Magazine - Quarter 4, 2017 - 12
BISA Magazine - Quarter 4, 2017 - 13
BISA Magazine - Quarter 4, 2017 - 14
BISA Magazine - Quarter 4, 2017 - 15
BISA Magazine - Quarter 4, 2017 - 16
BISA Magazine - Quarter 4, 2017 - 17
BISA Magazine - Quarter 4, 2017 - 18
BISA Magazine - Quarter 4, 2017 - 19
BISA Magazine - Quarter 4, 2017 - 20
BISA Magazine - Quarter 4, 2017 - 21
BISA Magazine - Quarter 4, 2017 - 22
BISA Magazine - Quarter 4, 2017 - 23
BISA Magazine - Quarter 4, 2017 - 24
BISA Magazine - Quarter 4, 2017 - 25
BISA Magazine - Quarter 4, 2017 - 26
BISA Magazine - Quarter 4, 2017 - 27
BISA Magazine - Quarter 4, 2017 - 28
BISA Magazine - Quarter 4, 2017 - 29
BISA Magazine - Quarter 4, 2017 - 30
BISA Magazine - Quarter 4, 2017 - 31
BISA Magazine - Quarter 4, 2017 - 32
BISA Magazine - Quarter 4, 2017 - Cover3
BISA Magazine - Quarter 4, 2017 - Cover4
https://www.nxtbook.com/nxtbooks/bisa/2017q4
https://www.nxtbook.com/nxtbooks/bisa/2017q3
https://www.nxtbook.com/nxtbooks/bisa/2017q2
https://www.nxtbook.com/nxtbooks/bisa/2017q1
https://www.nxtbook.com/nxtbooks/bisa/2016q4
https://www.nxtbook.com/nxtbooks/bisa/2016q3
https://www.nxtbook.com/nxtbooks/bisa/2016q2
https://www.nxtbook.com/nxtbooks/bisa/2016q1
https://www.nxtbook.com/nxtbooks/bisa/2015q4
https://www.nxtbook.com/nxtbooks/bisa/2015q3
https://www.nxtbook.com/nxtbooks/bisa/2015q2
https://www.nxtbook.com/nxtbooks/bisa/2015q1
https://www.nxtbook.com/nxtbooks/bisa/2014q4
https://www.nxtbook.com/nxtbooks/bisa/2014q3
https://www.nxtbook.com/nxtbooks/bisa/2014q2
https://www.nxtbook.com/nxtbooks/bisa/2014q1
https://www.nxtbookmedia.com