Morningstar Magazine - April/May 2018 - 47

6
Correlation = 87%
0
EXHIBIT 1

An Appealing Butter Effect... On the
surface, the butter production of Brazil
is correlated with levels of the S&P/TSX
Composite Index.

Annual Levels and Fitted
ValuesComposite
Using Annual
SP/TSX
Butter Production40%
in Brazil

Annual
Change:
1994-2017
S&P/TSX
Composite:
1994-2017

EXHIBIT 2
1994

2002

2010

...Slips Away Taking percentage rates
of change gives a different picture.

Annual Change: 1994-2017

S&P/TSX Composite

18K

40%

Fit

period, how long might he or she need to wait for
the fund to catch up with the benchmark? In other
words, how long can a good fund underperform?

20
12

Actual
-20
Correlation = 5%
-15
-10
-5
1994
2002
Butter
Production in Brazil

Correlation = 87%
-40
5
10
2010

Source: Morningstar Direct, U.S. Department of Agriculture,
author's calculations.

40%

-20

-20
Correlation = 5%

Zoo of Factors
There are so many factors to choose from that
Annual Change: 1994-2017
S&P/TSX Composite
John Cochrane of the University of Chicago coined
40%
the term "zoo of factors."5 Given this zoo of
hundreds of factors, anyone with the right data set,
a computer, and some programming knowledge
20
could back-test any number of factor-based
strategies in short order and report only the
favorable results. (See positive results bias.) As the0
late Nobel-prize winning economist Ronald H.
Coase said, "If you torture the data long enough,
it will confess."6
-20

-40
-15
-10
-5
Butter Production in Brazil

0.3

Source: Morningstar Direct, U.S. Department of Agriculture,
author's calculations.

No Controlled Experiments
In the hard sciences, empirical work mainly
consists of running controlled experiments in
which the effects of factors other than the variable
being studied are minimized or eliminated.
18000
Unfortunately, economists generally cannot
perform controlled experiments.7 Instead, they do
statistical analysis on historical data, with the
assumption that the underlying processes
that generate the data remained the same over
time. This is the assumption of stationarity.
Back-testing is an example of this kind of analysis.

Correlation = 5%
-40

Simulation, Not Reality
-40
-15
-10 of a back
-5 test 0is to see how
5 a strategy
10
The purpose
Butter
in Brazil in the past. But without
wouldProduction
have performed
Mr. Peabody's Wayback Machine, we can never
know for sure how it would have done. Some back
tests try to be more realistic by including
assumptions such as trading costs, but many
do not. But no matter what assumptions are made,
a back test is a simulation, not a historical fact.

Waiting for Godot

My colleague Maciej Kowara and I sought an
answer using historical returns on actively run
mutual funds and by running Monte Carlo
simulation. We collected returns on thousands of
funds globally from January 2003 to December
2017. We sorted these funds by 15-year return
in excess of the fund-specific benchmark. Looking
at the top 10% of funds by excess return,
we found that investors might have endured on
average almost six years of underperformance.
That's a level of patience many investors
are unlikely to possess. Our Monte Carlo results
confirm the finding that investors in the best of
funds may need great levels of patience.
0.4

0.4

Then, there are statistics that investors interpret0.3
properly, but that tell only part of the story.
0.2
Suppose that we can establish beyond the shadow
0.1
of a doubt that a fund that follows a given strategy
0.0
will outperform its
benchmark
long run.0.00
-0.15
-0.10over the
-0.05
-0.1
Even such a fund will have subperiods of underperformance. If an investor has the misfortune -0.2
of
Fit
investing in a fund at the beginning of such a -0.3

What we learned from this exercise was that even
0.2
if performance statistics are done correctly
and interpreted properly, they only tell half the 0.1
0.0
story. The lengths-0.15
of periods
of underperformance
-0.10
-0.05
0.00
are part of the other half. While funds that -0.1
should outperform their benchmarks over the long
-0.2
run may be desirable,
Fit they might not be suitable
-0.3
for investors with limited patience.
-0.4

Beware Statistics

It is all too easy to calculate nonsensical statistics,
such as the correlation between a stock market
index and butter production. Furthermore,
statistical analysis, in the absence of economic
analysis, can be misused to demonstrate
almost anything, as can happen with back tests
that have no linkage to economic reason.
Finally, using properly constructed statistics, but
in the absence of other relevant facts, can lead
to poor decisions, such as recommending a fund to
an investor who lacks the patience to hold it
through the rough patches. By paying careful
attention to these issues, we can prevent statistics
from
0.05 being the third kind of lie. K
Paul D. Kaplan, Ph.D., CFA, is director of research with
Morningstar Canada. He is a member of the editorial board
of Morningstar magazine.

-0.4
5 Cochrane coined this term in his 2011 presidential
18000 address to the American Finance Association. See Hsu, J. and Kalesnik, V. 2014. "Finding Smart Beta in the Factor Zoo," Fundamentals,
Research Affiliates, July.
6 Coase, R.H. 1995. Essays on Economics and Economists, University of Chicago Press.
7 A notable exception is the field of experimental economics where individual and group behavior is studied in laboratory settings.

global.morningstar.com/Morningstarmagazine

0.05

http://global.morningstar.com/Morningstarmagazine

Table of Contents for the Digital Edition of Morningstar Magazine - April/May 2018