Morningstar Magazine - April/May 2018 - 46

Strategies

Lies, Damn Lies, and Statistics
Common statistics hold plenty of pitfalls
for investors.

QUANT U

Paul D. Kaplan

Quoting Benjamin Disraeli, Mark Twain famously
quipped, "There are three kinds of lies: lies,
damned lies, and statistics." In the field of
investments, in which we rely heavily on statistical
analysis to evaluate the merits of investment
strategies and products, Twain's point is all too
relevant. In this edition of Quant U, I discuss
the pitfalls of some of the most common statistical
analysis done in the investment industry.
Correlation Is Not Causation1

One statistic that is all too easy to be misleading
is correlation, starting with its definition. How
many times have we heard that correlation
measures the tendency for two variables to move
up and down together? That's not quite right.
What correlation actually measures is the degree
to which two variables, each in excess of its
own average, are statistically related.
The other major mistake often made with respect
to correlation is causation. Seeing that two
variables are statistically related, we too easily
jump to the conclusion that there is a causal
relationship between them. But correlation and
causation are two very different things.
As mentioned in a Forbes article a couple of
years ago,2 academic David Leinweber drove this
point home in a paper in the mid-1990s that
showed that there was a very high correlation

between the annual level of the S&P 500
and the annual production of butter in Bangladesh.
The author presented the results for the period
1981 through 1993 and found that the correlation
over this period was about 87%.3
I wanted to see if I could find a similar correlation
for the S&P/TSX Composite over a more recent
period. It didn't take me long to discover that for
the Canadian stock market, it's the butter
production of Brazil from 1994 through 2017 that
does the trick. In EXHIB IT 1 , I draw the level
of the S&P/TSX Composite for each year as a red
circle and a blue line that shows the level
of the S&P/TSX Composite predicted by the annual
butter production in Brazil. They appear to be
strongly related. As in Leinweber's example, the
correlation is about 87%.
If you are thinking that there must be some
trick to finding dairy production numbers that are
correlated with stock market indexes, you're
right. The trick is to use trended variables. Over
any period of time, if two variables are trending
upward, such as a stock market index and
production in a growing dairy industry, they are
positively correlated, even if there is no causal link
between them.
The solution to trended variables is to remove
the trends. With both stock market indexes
and production levels, the natural way to remove
the trends is to take the percentage rate of
change of each variable. I did that for the annual
levels of the S&P/TSX Composite and annual

Brazilian butter production. EXHIBIT 2 plots the
annual percentage rates of change of both of
these variables. Now, we get the expected result
of almost no correlation (just an insignificant 5%).
But even if we have constructed the variables
properly, correlation is still not causation.
If A and B are correlated, it could be that there
is a third variable, C, related to both of them,
that we cannot observe.
When trying to find causation, one must
look to economic reasoning, not just statistical
links. This is especially important to keep
in mind when evaluating quantitative investment
strategies, especially those behind new
strategic-beta exchange-traded funds. Any causal
explanation must be made apart from
the statistics.
Watch Your Back (Test)

Correlation is not the only statistic in which
statistical significance can get confused
with causation. A common statistical procedure
in investment management is back-testing. Back
tests are run because the period of live
performance is often quite limited, nonexistent if
the strategy has yet to go live. The idea
is that if a strategy back-tests well, it should do
well in real time. But this can only be the case
if there are causal links between what the strategy
does at each point in time and its subsequent
performance. There needs to be an economic
rationale for the strategy before back-testing it.
There are several issues that should be
considered when evaluating a back test, especially
one involving a factor-based strategy:
Positive Results Bias
As my colleague Ben Johnson says, "There
is no such thing as a bad-looking back test."4
When we are presented with impressive
back-test results, we don't know how many other
strategies or factors were tried that didn't
turn out very well.

1 This section is taken from the script of a video report that I did for Morningstar.ca called "Correlation Is Not Causation," Feb. 13, 2018.
2 Sizemore, C. 2015. "The Bangladesh Butter Indicator Says Buy!" Forbes, March 4. I thank Larry Siegel for bringing this article to my attention.
3 Leinweber, D.J. 1995. "Stupid Data Miner Tricks: Overfitting the S&P 500," California Institute of Technology working paper.
4 "Be Wary of Strategic-Beta ETFs with Rosy Back-Tested Results," Morningstar.ca video report, April 27, 2017.

46

Morningstar April/May 2018


http://www.Morningstar.ca http://www.Morningstar.ca

Table of Contents for the Digital Edition of Morningstar Magazine - April/May 2018

Contents
Morningstar Magazine - April/May 2018 - Cover1
Morningstar Magazine - April/May 2018 - Cover2
Morningstar Magazine - April/May 2018 - 1
Morningstar Magazine - April/May 2018 - 2
Morningstar Magazine - April/May 2018 - Contents
Morningstar Magazine - April/May 2018 - 4
Morningstar Magazine - April/May 2018 - 5
Morningstar Magazine - April/May 2018 - 6
Morningstar Magazine - April/May 2018 - 7
Morningstar Magazine - April/May 2018 - 8
Morningstar Magazine - April/May 2018 - 9
Morningstar Magazine - April/May 2018 - 10
Morningstar Magazine - April/May 2018 - 11
Morningstar Magazine - April/May 2018 - 12
Morningstar Magazine - April/May 2018 - 13
Morningstar Magazine - April/May 2018 - 14
Morningstar Magazine - April/May 2018 - 15
Morningstar Magazine - April/May 2018 - 16
Morningstar Magazine - April/May 2018 - 17
Morningstar Magazine - April/May 2018 - 18
Morningstar Magazine - April/May 2018 - 19
Morningstar Magazine - April/May 2018 - 20
Morningstar Magazine - April/May 2018 - 21
Morningstar Magazine - April/May 2018 - 22
Morningstar Magazine - April/May 2018 - 23
Morningstar Magazine - April/May 2018 - 24
Morningstar Magazine - April/May 2018 - 25
Morningstar Magazine - April/May 2018 - 26
Morningstar Magazine - April/May 2018 - 27
Morningstar Magazine - April/May 2018 - 28
Morningstar Magazine - April/May 2018 - 29
Morningstar Magazine - April/May 2018 - 30
Morningstar Magazine - April/May 2018 - 31
Morningstar Magazine - April/May 2018 - 32
Morningstar Magazine - April/May 2018 - 33
Morningstar Magazine - April/May 2018 - 34
Morningstar Magazine - April/May 2018 - 35
Morningstar Magazine - April/May 2018 - 36
Morningstar Magazine - April/May 2018 - 37
Morningstar Magazine - April/May 2018 - 38
Morningstar Magazine - April/May 2018 - 39
Morningstar Magazine - April/May 2018 - 40
Morningstar Magazine - April/May 2018 - 41
Morningstar Magazine - April/May 2018 - 42
Morningstar Magazine - April/May 2018 - 43
Morningstar Magazine - April/May 2018 - 44
Morningstar Magazine - April/May 2018 - 45
Morningstar Magazine - April/May 2018 - 46
Morningstar Magazine - April/May 2018 - 47
Morningstar Magazine - April/May 2018 - 48
Morningstar Magazine - April/May 2018 - 49
Morningstar Magazine - April/May 2018 - 50
Morningstar Magazine - April/May 2018 - 51
Morningstar Magazine - April/May 2018 - 52
Morningstar Magazine - April/May 2018 - 53
Morningstar Magazine - April/May 2018 - 54
Morningstar Magazine - April/May 2018 - 55
Morningstar Magazine - April/May 2018 - 56
Morningstar Magazine - April/May 2018 - 57
Morningstar Magazine - April/May 2018 - 58
Morningstar Magazine - April/May 2018 - 59
Morningstar Magazine - April/May 2018 - 60
Morningstar Magazine - April/May 2018 - 61
Morningstar Magazine - April/May 2018 - 62
Morningstar Magazine - April/May 2018 - 63
Morningstar Magazine - April/May 2018 - 64
Morningstar Magazine - April/May 2018 - 65
Morningstar Magazine - April/May 2018 - 66
Morningstar Magazine - April/May 2018 - 67
Morningstar Magazine - April/May 2018 - 68
Morningstar Magazine - April/May 2018 - 69
Morningstar Magazine - April/May 2018 - 70
Morningstar Magazine - April/May 2018 - 71
Morningstar Magazine - April/May 2018 - 72
Morningstar Magazine - April/May 2018 - 73
Morningstar Magazine - April/May 2018 - 74
Morningstar Magazine - April/May 2018 - 75
Morningstar Magazine - April/May 2018 - 76
Morningstar Magazine - April/May 2018 - 77
Morningstar Magazine - April/May 2018 - 78
Morningstar Magazine - April/May 2018 - 79
Morningstar Magazine - April/May 2018 - 80
Morningstar Magazine - April/May 2018 - Cover3
Morningstar Magazine - April/May 2018 - Cover4
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2021q1
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q4
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q3
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q2
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2020q1
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019winter
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019fall
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019summer
https://www.nxtbook.com/nxtbooks/morningstar/magazine_2019spring
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20191201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20181011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20180203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20181201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20171011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20170203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20171201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20161011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20160203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20161201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20151011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20150203
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20151201
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20141011
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140809
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140607
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140405
https://www.nxtbook.com/nxtbooks/morningstar/magazine_20140203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20141201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20131011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20130203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20131201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20121011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120405
https://www.nxtbook.com/nxtbooks/morningstar/investorconference2012
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20120203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20121201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20111011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20110203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20111201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20101011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100809_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100607_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100405_lincoln
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20100203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20101201
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20091011
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090809
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090607
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090405
https://www.nxtbook.com/nxtbooks/morningstar/advisor_20090203
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008fall
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008summer
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007spring
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007fall
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2007summer
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008spring
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008catalog
https://www.nxtbook.com/nxtbooks/morningstar/advisor_2008winter
https://www.nxtbookmedia.com