Morningstar Magazine - April/May 2018 - 47

6
Correlation = 87%
0
EXHIBIT 1

An Appealing Butter Effect... On the
surface, the butter production of Brazil
is correlated with levels of the S&P/TSX
Composite Index.

Annual Levels and Fitted
ValuesComposite
Using Annual
SP/TSX
Butter Production40%
in Brazil

Annual
Change:
1994-2017
S&P/TSX
Composite:
1994-2017

EXHIBIT 2
1994

2002

2010

...Slips Away Taking percentage rates
of change gives a different picture.

Annual Change: 1994-2017

S&P/TSX Composite

18K

40%

20

Fit

period, how long might he or she need to wait for
the fund to catch up with the benchmark? In other
words, how long can a good fund underperform?

20
12

0

0

Actual
-20
Correlation = 5%
-15
-10
-5
1994
2002
Butter
Production in Brazil

0

Correlation = 87%
-40
5
10
2010

6

0

Source: Morningstar Direct, U.S. Department of Agriculture,
author's calculations.

40%

20

0

-20

-20
Correlation = 5%

Zoo of Factors
There are so many factors to choose from that
Annual Change: 1994-2017
S&P/TSX Composite
John Cochrane of the University of Chicago coined
40%
the term "zoo of factors."5 Given this zoo of
hundreds of factors, anyone with the right data set,
a computer, and some programming knowledge
20
could back-test any number of factor-based
strategies in short order and report only the
favorable results. (See positive results bias.) As the0
late Nobel-prize winning economist Ronald H.
Coase said, "If you torture the data long enough,
it will confess."6
-20

-40
-15
-10
-5
Butter Production in Brazil

0

5

10

0.3

Source: Morningstar Direct, U.S. Department of Agriculture,
author's calculations.

No Controlled Experiments
In the hard sciences, empirical work mainly
consists of running controlled experiments in
which the effects of factors other than the variable
being studied are minimized or eliminated.
18000
Unfortunately, economists generally cannot
perform controlled experiments.7 Instead, they do
statistical analysis on historical data, with the
assumption that the underlying processes
that generate the data remained the same over
time. This is the assumption of stationarity.
Back-testing is an example of this kind of analysis.

Correlation = 5%
-40

Simulation, Not Reality
-40
-15
-10 of a back
-5 test 0is to see how
5 a strategy
10
The purpose
Butter
in Brazil in the past. But without
wouldProduction
have performed
Mr. Peabody's Wayback Machine, we can never
know for sure how it would have done. Some back
tests try to be more realistic by including
assumptions such as trading costs, but many
do not. But no matter what assumptions are made,
a back test is a simulation, not a historical fact.

Waiting for Godot

My colleague Maciej Kowara and I sought an
answer using historical returns on actively run
mutual funds and by running Monte Carlo
simulation. We collected returns on thousands of
funds globally from January 2003 to December
2017. We sorted these funds by 15-year return
in excess of the fund-specific benchmark. Looking
at the top 10% of funds by excess return,
we found that investors might have endured on
average almost six years of underperformance.
That's a level of patience many investors
are unlikely to possess. Our Monte Carlo results
confirm the finding that investors in the best of
funds may need great levels of patience.
0.4

0.4

Then, there are statistics that investors interpret0.3
properly, but that tell only part of the story.
0.2
Suppose that we can establish beyond the shadow
0.1
of a doubt that a fund that follows a given strategy
0.0
will outperform its
benchmark
long run.0.00
-0.15
-0.10over the
-0.05
-0.1
Even such a fund will have subperiods of underperformance. If an investor has the misfortune -0.2
of
Fit
investing in a fund at the beginning of such a -0.3

What we learned from this exercise was that even
0.2
if performance statistics are done correctly
and interpreted properly, they only tell half the 0.1
0.0
story. The lengths-0.15
of periods
of underperformance
-0.10
-0.05
0.00
are part of the other half. While funds that -0.1
should outperform their benchmarks over the long
-0.2
run may be desirable,
Fit they might not be suitable
-0.3
for investors with limited patience.
-0.4

Beware Statistics

It is all too easy to calculate nonsensical statistics,
such as the correlation between a stock market
index and butter production. Furthermore,
statistical analysis, in the absence of economic
analysis, can be misused to demonstrate
almost anything, as can happen with back tests
that have no linkage to economic reason.
Finally, using properly constructed statistics, but
in the absence of other relevant facts, can lead
to poor decisions, such as recommending a fund to
an investor who lacks the patience to hold it
through the rough patches. By paying careful
attention to these issues, we can prevent statistics
from
0.05 being the third kind of lie. K
Paul D. Kaplan, Ph.D., CFA, is director of research with
Morningstar Canada. He is a member of the editorial board
of Morningstar magazine.

-0.4
5 Cochrane coined this term in his 2011 presidential
18000 address to the American Finance Association. See Hsu, J. and Kalesnik, V. 2014. "Finding Smart Beta in the Factor Zoo," Fundamentals,
Research Affiliates, July.
6 Coase, R.H. 1995. Essays on Economics and Economists, University of Chicago Press.
7 A notable exception is the field of experimental economics where individual and group behavior is studied in laboratory settings.

global.morningstar.com/Morningstarmagazine

47

0.05


http://global.morningstar.com/Morningstarmagazine

Table of Contents for the Digital Edition of Morningstar Magazine - April/May 2018

Contents
Morningstar Magazine - April/May 2018 - Cover1
Morningstar Magazine - April/May 2018 - Cover2
Morningstar Magazine - April/May 2018 - 1
Morningstar Magazine - April/May 2018 - 2
Morningstar Magazine - April/May 2018 - Contents
Morningstar Magazine - April/May 2018 - 4
Morningstar Magazine - April/May 2018 - 5
Morningstar Magazine - April/May 2018 - 6
Morningstar Magazine - April/May 2018 - 7
Morningstar Magazine - April/May 2018 - 8
Morningstar Magazine - April/May 2018 - 9
Morningstar Magazine - April/May 2018 - 10
Morningstar Magazine - April/May 2018 - 11
Morningstar Magazine - April/May 2018 - 12
Morningstar Magazine - April/May 2018 - 13
Morningstar Magazine - April/May 2018 - 14
Morningstar Magazine - April/May 2018 - 15
Morningstar Magazine - April/May 2018 - 16
Morningstar Magazine - April/May 2018 - 17
Morningstar Magazine - April/May 2018 - 18
Morningstar Magazine - April/May 2018 - 19
Morningstar Magazine - April/May 2018 - 20
Morningstar Magazine - April/May 2018 - 21
Morningstar Magazine - April/May 2018 - 22
Morningstar Magazine - April/May 2018 - 23
Morningstar Magazine - April/May 2018 - 24
Morningstar Magazine - April/May 2018 - 25
Morningstar Magazine - April/May 2018 - 26
Morningstar Magazine - April/May 2018 - 27
Morningstar Magazine - April/May 2018 - 28
Morningstar Magazine - April/May 2018 - 29
Morningstar Magazine - April/May 2018 - 30
Morningstar Magazine - April/May 2018 - 31
Morningstar Magazine - April/May 2018 - 32
Morningstar Magazine - April/May 2018 - 33
Morningstar Magazine - April/May 2018 - 34
Morningstar Magazine - April/May 2018 - 35
Morningstar Magazine - April/May 2018 - 36
Morningstar Magazine - April/May 2018 - 37
Morningstar Magazine - April/May 2018 - 38
Morningstar Magazine - April/May 2018 - 39
Morningstar Magazine - April/May 2018 - 40
Morningstar Magazine - April/May 2018 - 41
Morningstar Magazine - April/May 2018 - 42
Morningstar Magazine - April/May 2018 - 43
Morningstar Magazine - April/May 2018 - 44
Morningstar Magazine - April/May 2018 - 45
Morningstar Magazine - April/May 2018 - 46
Morningstar Magazine - April/May 2018 - 47
Morningstar Magazine - April/May 2018 - 48
Morningstar Magazine - April/May 2018 - 49
Morningstar Magazine - April/May 2018 - 50
Morningstar Magazine - April/May 2018 - 51
Morningstar Magazine - April/May 2018 - 52
Morningstar Magazine - April/May 2018 - 53
Morningstar Magazine - April/May 2018 - 54
Morningstar Magazine - April/May 2018 - 55
Morningstar Magazine - April/May 2018 - 56
Morningstar Magazine - April/May 2018 - 57
Morningstar Magazine - April/May 2018 - 58
Morningstar Magazine - April/May 2018 - 59
Morningstar Magazine - April/May 2018 - 60
Morningstar Magazine - April/May 2018 - 61
Morningstar Magazine - April/May 2018 - 62
Morningstar Magazine - April/May 2018 - 63
Morningstar Magazine - April/May 2018 - 64
Morningstar Magazine - April/May 2018 - 65
Morningstar Magazine - April/May 2018 - 66
Morningstar Magazine - April/May 2018 - 67
Morningstar Magazine - April/May 2018 - 68
Morningstar Magazine - April/May 2018 - 69
Morningstar Magazine - April/May 2018 - 70
Morningstar Magazine - April/May 2018 - 71
Morningstar Magazine - April/May 2018 - 72
Morningstar Magazine - April/May 2018 - 73
Morningstar Magazine - April/May 2018 - 74
Morningstar Magazine - April/May 2018 - 75
Morningstar Magazine - April/May 2018 - 76
Morningstar Magazine - April/May 2018 - 77
Morningstar Magazine - April/May 2018 - 78
Morningstar Magazine - April/May 2018 - 79
Morningstar Magazine - April/May 2018 - 80
Morningstar Magazine - April/May 2018 - Cover3
Morningstar Magazine - April/May 2018 - Cover4
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2021q1
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q4
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q3
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q2
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q1
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019winter
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019fall
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019summer
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019spring
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20191201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20181011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20181201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20171011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20171201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20161011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20161201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20151011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20151201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20141011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20141201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20131011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20131201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20121011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120405
https://www.nxtbook.com/nxtbooks/morningstar/investorconference2012
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20121201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20111011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20111201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20101011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100809_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100607_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100405_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20101201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20091011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008fall
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008summer
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007spring
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007fall
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007summer
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008spring
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008catalog
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008winter
https://www.nxtbookmedia.com