Survivorship Bias - the single greatest fallacy in investing

[Dodge]

Survivorship bias should not be overlooked, in my opinion it is the single greatest fallacy in investing.

It revolves around the tendency for failed companies to be excluded from performance studies due to the fact that they no longer exist. Survivorship bias causes the results of studies to skew higher because only companies which were successful enough to survive until the end of the period are included. Similarly, mutual fund performance may be misleading due to Survivorship bias when underperforming funds are merged or discontinued.

Here's an incredible article which vividly explains the pitfalls of Survivorship bias, and not just in investing.  While I'm sure most here won't spend the 15 minutes or so reading it, I'd rather not spoil the stories!  I'll just give the opener as a teaser:



http://youarenotsosmart.com/2013/05/23/survivorship-bias/

Here's what survivorship bias looks like:



Source: http://www.traderplanet.com/articles/view/163387-improve-market-breadth-analysis-survivorship-bias/

[forummm]

Thanks for the interesting post. Is the last graph a total return?

[forummm]

How do you eliminate survivorship bias from the Morningstar graph? What about AIG, Fannie Mae, etc?

[691175002]

To be completely technical, the dividend aristocrat charts are an example of look-ahead bias, not survivorship bias.  The hypothetical investments are made using information not available at the time (ex: Purchase the holdings of the 2014 index in 2005 or earlier).

Survivorship bias is a result of incomplete data.

This would have been impossible to actually do 25 years ago, because the 1989 version of us can't know what companies will be on the list in 2015!

DEFINITION OF 'LOOK-AHEAD BIAS'
Bias created by the use of information or data in a study or simulation that would not have been known or available during the period being analyzed. This will usually lead to inaccurate results in the study or simulation.

[Dodge]

To be completely technical, the dividend aristocrat charts are an example of look-ahead bias, not survivorship bias.  The hypothetical investments are made using information not available at the time (ex: Purchase the holdings of the 2014 index in 2005 or earlier).

Survivorship bias is a result of incomplete data.

This would have been impossible to actually do 25 years ago, because the 1989 version of us can't know what companies will be on the list in 2015!

DEFINITION OF 'LOOK-AHEAD BIAS'
Bias created by the use of information or data in a study or simulation that would not have been known or available during the period being analyzed. This will usually lead to inaccurate results in the study or simulation.

Interesting!  Thanks for that.  I interpreted it as a study of which current companies had the biggest dividend yield in 1989, then seeing what would have happened if you invested in those companies from 1989-2015.  Of course this excludes all the companies which had high dividend yield in 1989, then went out of business in 1990.  But you are correct, the way I wrote it, it fits the Look-Ahead Bias.  Interesting that both can lead to the same error.

[Dodge]

How do you eliminate survivorship bias from the Morningstar graph? What about AIG, Fannie Mae, etc?

Yes, the last graph is a total return graph.  You don't have to eliminate survivorship bias from the last graph, because the last graph is actual performance data.  There is no survivorship bias to eliminate.  It shows you how the actual S&P500, Total Stock Index Fund, and Dividend Aristocrat funds performed during these periods, including the fall of AIG and Fannie Mae.

[index]

I interpreted it as a study of which current companies had the biggest dividend yield in 1989, then seeing what would have happened if you invested in those companies from 1989-2015.  Of course this excludes all the companies which had high dividend yield in 1989, then went out of business in 1990.  But you are correct, the way I wrote it, it fits the Look-Ahead Bias.  Interesting that both can lead to the same error.

All indexes demonstrate survivorship bias. Your example just demonstrates the performance of two indexes, the Dividend Aristocrats vs the S&P 500, that use  different inclusion criteria..

S&P 500® Dividend Aristocrats® measure the performance S&P 500 companies that have increased dividends every year for the last 25 consecutive years. The Index treats each constituent as a distinct investment opportunity without regard to its size by equally weighting each company

So the dividend aristocrat index equally weights all S&P 500 companies that have raised dividends 25 consecutive years.

This strategy is easy to back test because it has easy selection criteria. With a little work, you could look at dividend history and the constituents of the S&P 500 in 1989 and create the index, changing it every year as companies with 25+ years of dividend growth were added and dropped from the S&P.

Many tend to think of indexes as static collections of stocks when, in fact, they are dynamic portfolios in and of themselves. Look at the S&P 500 in 1999 and compared it with the S&P 500 of today.

http://etfdb.com/history-of-the-s-and-p-500/#1999

In 1999 vs 2013 the S&P 500's top 10 holdings were:

1. Microsoft - Apple
2. GE - Exxon
3. Cisco - Google
4. Walmart - Microsoft
5. Exxon - GE
6. Intel - JNJ
7. Lucent - Chevron
8. IBM - P&G
9. Citi - JP Morgan Chase
10. AOL - Wells Fargo

So of the top 10 in 1999 to 2013, only 3 overlap. AOL and Lucent are not even around anymore!

Taking an index and tracking the constituents back a number of years would demonstrate exercising survivorship bias but the example of tracking returns of the index over time or comparing two indicies is not.

An interesting question would be why the Dividend Aristocrats out performs the S&P 500... 

 

[Dodge]

Taking an index and tracking the constituents back a number of years would demonstrate exercising survivorship bias but the example of tracking returns of the index over time or comparing two indicies is not.

Correct.  The first three charts exhibited survivorship bias (or maybe look-ahead bias, depending on their thought process), while the fourth chart which just tracks the returns of the index, does not exhibit survivorship bias.

The purpose of this thread isn't to debate the merits of dividend investing.  The purpose of showing the three charts, is to highlight the difference between the first three (which exhibit a bias), and the fourth (which does not).

[tj]

A legitimate concern.

Dodge: Are you 100% VTWSX? If not, why not?

[Dodge]

A legitimate concern.

Dodge: Are you 100% VTWSX? If not, why not?

My personal holdings are not relevant to the thread.

[innerscorecard]

Survivorship bias can be overcome by using a "point in time" database such as Compustat. These databases are expensive, of course, due to how much work is involved in putting them together.

[deborah]

The last graph is not linear like the others - is there any way you can have a linear one?

[index]

First off Dodge. Your first chart shows the S&P vs SDY using price only. It is also exactly backwards!



Here is the total return chart without the log scale of the S&P vs SDY:



I pose the question to you again? Why did SDY beat the S&P 500? I'll give you a clue, SDY uses superior inclusion criteria. So what about the inclusion criteria makes SDY better than VFIAX or VTSAX?

[Dodge]

I pose the question to you again? Why did SDY beat the S&P 500? I'll give you a clue, SDY uses superior inclusion criteria. So what about the inclusion criteria makes SDY better than VFIAX or VTSAX?

Feel free to discuss the merits of dividend investing here, where I've posted the same charts for discussion:

dividend-portfolio-vs-index-fund

[index]

I pose the question to you again? Why did SDY beat the S&P 500? I'll give you a clue, SDY uses superior inclusion criteria. So what about the inclusion criteria makes SDY better than VFIAX or VTSAX?

Feel free to discuss the merits of dividend investing here, where I've posted the same charts for discussion:

dividend-portfolio-vs-index-fund

I don't think dividends have anything to do with it.

Also I couldn't justify the differences between SDY and the S&P 500. I looked at bloomberg (the source) and discovered SDY is not even the a Dividend Aristocrat index! NOBL only has 53 holdings.

The correct Div Aristocrat ETF is NOBL. SDY tracks the S&P High Yield Dividend Aristocrats Index. The Index is designed to measure the performance of the 60 highest dividend yielding S&P Composite 1500 Index constituents (current holdings 101) that have followed a managed-dividends policy of consistently increasing dividends every year for at least 25 consecutive years. 

This chart (source 20k/yr Bloomberg Terminal) is indeed correct:



So I ask you again. Why does this index beat the S&P? I'm not looking for a discussion on the merits of dividend investing. That is not the reason.

[Dodge]

I considered adding NOBL to the original post, but it doesn't have much history.  It's there now, as another example of the actual returns of a dividend aristocrats index not matching the survivorship bias charts.

[uwp]

Not only has SDY beat the S&P, it did it with a lower standard deviation, and a .35% expense ratio drag.

[skyrefuge]

This chart (source 20k/yr Bloomberg Terminal) is indeed correct:

Ok, so what exactly is that chart showing? Here's what I know:

- The S&P 500 Dividend Aristocrats Index (SPDAUDP) was created May 2, 2005. Dodge's first chart in his first post actually shows the total return of that index vs. the total return of the S&P 500 since that date. It is taken from this S&P document (pdf).
- No funds seem to have tracked this index until NOBL was created, on October 9, 2013.

So your Bloomberg data then shows something that started in 1989. Do you have any idea what this is? A dynamic index created retroactively using the rules that were defined in 2005? A static group of the stocks that were contained in the index as of some date? Something else?

[index]

I considered adding NOBL to the original post, but it doesn't have much history.  It's there now, as another example of the actual returns of a dividend aristocrats index not matching the survivorship bias charts.

You still do not understand. Showing NOBL for the passed year is pointless and matches the Dividend Aristocrat Index with a little tracking error. 

This is what you did:

You looked at charts of the Dividend Aristocrat Index and said I bet I can show survivorship bias. These are your charts 1, 2, and 3.

You then went to Morningstar and put in a symbol for SDY (a completely different index) and said: Look this didn't perform the same! Survivorship bias!

This is akin to saying the etf SPY does not track the S&P well then using VTSAX as your S&P proxy when comparing two charts.

The S&P has point in time total return results for the Dividend Aristocrat Index (tracker: SPDAUDT) going back to 2005:

http://us.spindices.com/indices/strategy/sp-500-dividend-aristocrats

If you were to pay 20k/yr for a Bloomberg Terminal, you could get point in time performance data for SPDAUDT back to 1989. This is shown in my last chart, and your chart #2. No survivorship bias!

Here is NOBL (the ETF that actually tracks charts 1, 2, and 3 in your post) vs. the S&P 500, SPDAUDT, and SDY.



I pose again: Why does this index beat the S&P? I'm not looking for a discussion on the merits of dividend investing. That is not the reason.

[index]

This chart (source 20k/yr Bloomberg Terminal) is indeed correct:

Ok, so what exactly is that chart showing? Here's what I know:

- The S&P 500 Dividend Aristocrats Index (SPDAUDP) was created May 2, 2005. Dodge's first chart in his first post actually shows the total return of that index vs. the total return of the S&P 500 since that date. It is taken from this S&P document (pdf).
- No funds seem to have tracked this index until NOBL was created, on October 9, 2013.

So your Bloomberg data then shows something that started in 1989. Do you have any idea what this is? An dynamic index created retroactively using the rules that were defined in 2005? A static group of the stocks that were contained in the index as of some date? Something else?

The Bloomberg terminal actually uses the selection criteria for the index to recreate and back test the index. Meaning it looks at the S&P 500 in 1989 and says of those 500 companies, which have a history of dividend growth for the past 25 years? It adds those companies to the index. This is done every quarter from 1989. The back test goes back until 1989 because standardized quarterly data only started to be provided by Standard and Poors in 1964 with their launch of compustat. 1989 with 25 years of data takes your back to 1964.   

So yes this back test demonstrates a dynamic index created retroactively using the rules that were defined in 2005.

[sirdoug007]

Ok, so you found one random time period where SDY and presumably NOBL which you prefer but is too new did better than the S&P500.

Here is a six year period where VTI and SPY did better than SDY.  Where is the outperformance?

SDY in green
SPY in blue
VTI in orange

All total return from this website: http://www.etfreplay.com/charts.aspx

[Dodge]

Updated OP with a different example to try and keep the thread on topic.

[forummm]

The S&P has point in time total return results for the Dividend Aristocrat Index (tracker: SPDAUDT) going back to 2005:

http://us.spindices.com/indices/strategy/sp-500-dividend-aristocrats

<snip>

I pose again: Why does this index beat the S&P? I'm not looking for a discussion on the merits of dividend investing. That is not the reason.

The performance for the last 10 years is interesting. The DA dropped to below the 500 during the crash and has outpaced during the recovery/bull market.

If I had to guess, I would say that the search for yield in a historically low interest rate environment has driven the increased price performance of the DA. Utilities as a sector is an example of this too.

[sirdoug007]

I considered adding NOBL to the original post, but it doesn't have much history.  It's there now, as another example of the actual returns of a dividend aristocrats index not matching the survivorship bias charts.

You still do not understand. Showing NOBL for the passed year is pointless and matches the Dividend Aristocrat Index with a little tracking error. 

This is what you did:

You looked at charts of the Dividend Aristocrat Index and said I bet I can show survivorship bias. These are your charts 1, 2, and 3.

You then went to Morningstar and put in a symbol for SDY (a completely different index) and said: Look this didn't perform the same! Survivorship bias!

This is akin to saying the etf SPY does not track the S&P well then using VTSAX as your S&P proxy when comparing two charts.

The S&P has point in time total return results for the Dividend Aristocrat Index (tracker: SPDAUDT) going back to 2005:

http://us.spindices.com/indices/strategy/sp-500-dividend-aristocrats

If you were to pay 20k/yr for a Bloomberg Terminal, you could get point in time performance data for SPDAUDT back to 1989. This is shown in my last chart, and your chart #2. No survivorship bias!

Here is NOBL (the ETF that actually tracks charts 1, 2, and 3 in your post) vs. the S&P 500, SPDAUDT, and SDY.



I pose again: Why does this index beat the S&P? I'm not looking for a discussion on the merits of dividend investing. That is not the reason.

Why did NOBL beat the S&P500 by 0.85%?  Doesn't sound significant enough to care about to me. 

[Dodge]

Will repost at a later time without the dividend charts.

MOD NOTE: Thread restored.