The Money Mustache Community
Learning, Sharing, and Teaching => Investor Alley => Topic started by: Seadog on May 24, 2017, 05:39:46 AM

"You can't beat the index" has almost become a bit of a religion it seems to some people. Not saying I wholly disagree, but I'd like to know why exactly, backed up with some numbers.
My school of thought is that an individual, or random spattering of stocks bought and held should have a 50% chance of beating the market approaching market returns (decreasing variance) as you trend toward owning every stock in the index. Why wouldn't that be the case?
Then too it follows, that if stock picking truly is a losing game, then it seems to me the inverse would be a winning one. Namely, buying the entire index, *except* for a few "chosen" stocks. Since you can expect those chosen stocks to underperform the market, the remainder that you didn't "choose" would exceed it no? Similarly, play the same choosing game, but just go short since you can confidently say it will do worse than the market.
I simply can't reconcile how 90+% or whatever the quoted number is can underperform. Simply because when you look at a large enough sampling of funds, it would be essentially mimic the entire market no?

"You can't beat the index" has almost become a bit of a religion it seems to some people. Not saying I wholly disagree, but I'd like to know why exactly, backed up with some numbers.
My school of thought is that an individual, or random spattering of stocks bought and held should have a 50% chance of beating the market approaching market returns (decreasing variance) as you trend toward owning every stock in the index. Why wouldn't that be the case?
Then too it follows, that if stock picking truly is a losing game, then it seems to me the inverse would be a winning one. Namely, buying the entire index, *except* for a few "chosen" stocks. Since you can expect those chosen stocks to underperform the market, the remainder that you didn't "choose" would exceed it no? Similarly, play the same choosing game, but just go short since you can confidently say it will do worse than the market.
I simply can't reconcile how 90+% or whatever the quoted number is can underperform. Simply because when you look at a large enough sampling of funds, it would be essentially mimic the entire market no?
Of course you can beat the market. But can you do it constantly and with certainty?
Your example is good: you have a 50:50 chance to beat the market. So will it be financially wise to try to beat the market? The expected value of your bet is the return of the market, but you have to invest time and fees, so the real expected value is lower.

Agree with L.A.S. that the costs and then the market's overall efficiency explain most, maybe all, of what you're talking about.
For example, I think the stock market returned about a 11% nominal return in the ten years ending 2005. The top quartile return was 12% and the bottom quartile percent was 10%. So if an investor has a choice between getting that 11% or paying 1% to 2% to try to beat that 11%, it's pretty tough.
In fact, I think active investors basically, on average, fall short of the passive investor's index returns by their investment costs.
Oh, maybe one other thing: Those investment costs don't include just the fund expense ratios. Other costs like taxes and what I think Bogle refers to as market impact costs add up too.

It is definitely possible to beat the index, but it is hard. The main reason is that the performance of individual stocks is not subject to a gaussian distribution.
Imagine the S&P 500 to yield 10% in some year. A gaussian distrubition of gains for the individual stocks would mean that that the 250th best stock would hit exactly 10% gain, with the best stock maybe gaining 30% and the worst losing 10%.
That is usually not the case. The curve of individual gains is skewed and resembles a LogNormalDistibution https://en.wikipedia.org/wiki/Lognormal_distribution
That means stock number 250 of the S&P 500 will almost certainly be below 10% gain.
Most of the gains of the whole index comes from the top performers. This is typical for the long vs. short debate: potential upturn is almost infinte, potential downturn is limited to 100% but downturns are more likely. If top performers make real progress possible in their respective sector, the noninnovating crop fades away leaving the sector in a far better state than it was before.
So for a random subselection of this number of individual companies to be as good as the index or better, you have to pick some with stellar performance for all the more likely picking of boring lowaverage performers. So your knowledge and analysis has to be significant better than random. That is hard the hard part.
But I think it is possible to select the macroscopic growth markets. To me, that is technology and healthcare. And if you picked individual stocks, stick to your own consistend algorithm. Like "Pick after random drop, sell at x % plus"

All domestic equity funds v.s. the S&P 1500 ("total market"), 87% failed to beat the index.
http://us.spindices.com/documents/spiva/spivausmidyear2016.pdf
Other stats available there. SPIVA is very rigorous, even including funds that disappeared (avoids survivorship bias). The above figure is probably before calculating taxes, which also tend to favor passive indexing.

That is usually not the case. The curve of individual gains is skewed and resembles a LogNormalDistibution https://en.wikipedia.org/wiki/Lognormal_distribution
That means stock number 250 of the S&P 500 will almost certainly be below 10% gain.
Most of the gains of the whole index comes from the top performers.
I had thought of this, and decided not to mention it because I don't think it would make a difference based on random pickings. Even if only 10% of stocks are winners, you would expect to hit one with having 10 stocks, or again by picking only 98% of stocks in the index, that 2% you don't have would expect to fall in the losing 90%, thus beat the average.
All domestic equity funds v.s. the S&P 1500 ("total market"), 87% failed to beat the index.
http://us.spindices.com/documents/spiva/spivausmidyear2016.pdf
Again, I'm not saying I disagree, I just want to know why exactly. Is it purely a function of fees? You would assume that any large sampling of funds would mimic the market as a whole. If 87% fail to beat it, wouldn't merely doing the opposite of the typical fund beat it then? Or do those 13% that beat it beat it by such a huge margin to offset the other 87%?

As to active management failing to beat the index(s), I would say there are many factors other than fees that hold them back. They tend to beat the index(s)/market when it is down. I think it's because they are always hedging their gains. Conversely, I think it harms their performance when the market is up. The market is up many more years than it is down (5:1?). Additionally, all of the rules they have to follow such as not holding any individual equity > 5% of the portfolio, the large quantity of capital needing to be invested leads to having to decide between which equity is not worse than the other (even though they may both underperform), Window dressing, etc...

Of course you can beat the market. But can you do it constantly and with certainty?
This is exactly what I was going to say. It's ludicrous to say you CAN'T beat the market. Of course you CAN. But WILL you? Chances are, no.

If you want some math, check out the central limit theorem (https://en.wikipedia.org/wiki/Central_limit_theorem) and the law of large numbers (https://en.wikipedia.org/wiki/Law_of_large_numbers). You're right that a person should have a reasonable probability of beating the market if they pick their own stocks. Picking your own stocks for a year, it won't be all that uncommon to come out 15% ahead of the market, or 15% behind, for that matter.
But after more and more time doing this, the less likely it is that you'll beat the market by much. Someone who is doing all the right things and not making any behavioral mistakes should expect to get closer and closer to the market average the longer they invest. They may come out a few pennies ahead or a few pennies behind, but significant outperformance is extremely unlikely.

Seadog,
You're only 33 years old. You've only known the greatest bull market your entire investing life. Take time into account for your theories.
Pffst! Millennials! Lol.... kidding

Sharpe's law of active management is very precise and states that active investing is a negative sum game in aggregate. Specific investors can of course beat the market, but the average return of all investors cannot be higher than the index (since of course the index is the average).
Whether or not an investor can be consistantly skilled is an entirely different question and has to do with market efficiency. You can make an emperical case for persistant investor skill, but in theory and practice skilled investors tend to capture that extra return via fees so investing in a mutual or hedge fund has historically been a poor choice.
I'd be careful about applying the central limit theorem to investing since outcomes are not independent. For example is buying 30 small cap companies 30 seperate bets, or just one bet on small cap stocks? In practice investors tend to have a very specific style that goes in and out of favor over long cycles.
Finally, I will point out that even under perfectly efficient markets you can achieve any return, only the riskadjusted return is constrained. This is a very very critical point that people constantly ignore. Anyone can beat the market with extremely high probability if they are willing to take additional risk.
For example someone who is 110% invested has an extremely low probability of ruin and is expected to beat the overall market over time. Conversely, any investor can choose to go to cash or bonds and expect to underperform the market over time.
Note that in the market any system that consistently loses can be inversed to produce a winning strategy (although fees are a cost on both sides).

It's not empirical, but keep in mind that being able to consistently beat the market means you have some big advantage over millions of people.

Again, I'm not saying I disagree, I just want to know why exactly. Is it purely a function of fees? You would assume that any large sampling of funds would mimic the market as a whole. If 87% fail to beat it, wouldn't merely doing the opposite of the typical fund beat it then?
There are three major costs involved with active trading  manager expenses, taxes, and trading costs. Manager expenses are built into the expense ratio, but that's often small potatoes compared to the other two. Taxes are a big deal for active traders, as capital gains with every trade can quickly wipe out a large portion of your success on paper. And trading costs are the internal expenses that a fund incurs from trading (brokerage fees, market moves caused by the fund itself, bid/ask spreads, etc). Note that these trading costs are substantial and can be equivalent to several percent a year, and they are NOT included in the ER. The SEC only requires funds to report them in a difficulttointerpret line item on the fund SAI that nobody reads. Depending on the fund, internal trading costs can exceed the stated expense ratio by an order of magnitude!
Actively doing the opposite of what the average fund does will not make you more money because you still have the same taxes and trading costs. It has nothing to do with the markets and everything to do with the mechanics of active trading. The more you trade, the more it costs. That's why even in the index fund universe, it's important to seek out funds with the lowest amount of turnover. There's a lot more money leakage than just the ER.

Seadog,
You're only 33 years old. You've only known the greatest bull market your entire investing life. Take time into account for your theories.
Pffst! Millennials! Lol.... kidding
Sort of related to this point: Most people retired now haven't actually had that bad an outcome with their investing.
Even people who retired end of 2008 didn't, when you consider the two decades before that, end up with that bad a return.

Of course you can beat the market. But can you do it constantly and with certainty?
Your example is good: you have a 50:50 chance to beat the market. So will it be financially wise to try to beat the market? The expected value of your bet is the return of the market, but you have to invest time and fees, so the real expected value is lower.
My understanding is that you generally don't have a 50;50 chance to beat the market because of fees.
For active funds, I think it's something like 20% of funds beat the index after fees per year. But, I think that people who've looked at the data say that next year's winners aren't strongly correlated to next year's winners (i.e. beating the market is more luck than skill).
So, your chance of consistently beating the market over a 30 year period is: 0.2^(30)
That's a small number...it's why basically no one has done it.
If you're picking stocks yourself, you might be closer to 50:50. But it's still random, and at 50:50 you're still expecting similar returns as the market (it's probably less than that because you still probably pay transaction costs).
So, if the best you can expect in the long run is about the same as the market, why not just buy the market and use your time to do something else?

There is a lot of math that supports this. Start with the Markowitz portfolio theory which is commonly called modern portfolio theory. This theory basically posits that as you add risky assets, you can attain a similar return with lower variance. This can also be used to identify a portfolio that returns a greater return at the same variance. This extends to the idea that the optimal portfolio is a combination of the pool of all risky assets (the market return) and the risk free rate. Using these two, you can optimize the tradeoff between the variance of returns and the expected return. This material involves lots of linear algebra, but if you can understand least squares regression you will be able to follow it.
Pair with this the efficient market hypothesis and its various forms. Finally pair with that the work of Modigliani and miller that in essence identify the value of a firm as the combination of its earning power and the risk of its underlying assets. "The Cost of Capital, Corporation Finance and the Theory of Investment." This provided a foundation for equity pricing used by Markowitz.
I'm going to confess that this is off the top of my head and I could have gone fuzzy on some of the details since school. The topics above represent about 5 courses from my finance degree. You needn't go so deep for causal understanding, but you do need to know its not like there's a magic bullet answer of x+y means passive index funds rule.
Okay, so I reread your initial post and want to add...You are looking at things assuming there are no transaction costs (meaning the fees of the funds) associated with the portfolio of active funds. So if you take the market approximation by supplanting the managed funds for individual equities, you are correct that it should equal the market, but then you have to subtract the 12% fees from those funds. You end up with a similar frontier that runs just below the efficient frontier of the total market.

Here is some discussion around the skewed distribution of returns and the impact of this phenomenon on trying to pick winning stocks vs. buying the index.
https://www.bloomberg.com/news/articles/20170409/lopsidedstocksandthemathexplainingactivemanagerfutility

Excellent topic, a lot of great information and insights here.
A few questions.
Does anybody know the distribution of active traders by size and level of sophistication (for luck of a better term)? For example, 50% of active traders by volume of trades are institutions and 50% are individual traders. Ideally, I'd like to see % of hedge funds, banks, pension funds, sovereign wealth funds, active investment funds, other institutions, rich individuals who trade by themselves, smaller sophisticated individual traders (e.g, use their own algos) and "shoeshine boys who give stock tips". This would give a better idea as to the odds of success for a relatively sophisticated active trader be it an individual or an institution. For example, if half of active investments by volume are made by "shoeshine boys", the chances of success are still not too bad for somebody who can outsmart them. If 90% of volume is professional/institutional, not so much.
Related to previous question: quant/algo trading is becoming more popular. It requires access to complex and expensive data sources as well as expertise to process and make sense of them (e.g. big data technology, trading robots/algos, machine learning, etc). For example, quants use online sentiment data, geospatial data, credit card transaction data, etc, to understand how companies are doing before they release their financial reports. Given the investment and expertise required to do this, aren't good quantitative traders more likely to beat the market, particularly while they compete against less sophisticated investors (as long as quants are in the minority)? Case in point: Renaissance Technologies, a secretive hedge fund run by PhDs, believed to be the most successful hedge fund https://www.bloomberg.com/news/articles/20170425/renaissancemintsanotherbillionairewithtwomoreonthecusp
Suppose there are sophisticated institutional investors which are in a good position to consistently beat the market (at least most of the time). Can an individual active investor consistently outperform the market? For example, a genius math PhD who can create sophisticated trading algorithms and keep them a secret? Or does it require an institution with a group of people and deep pockets to buy data and infrastructure?
Note: I am using the terms "active investors" and "traders" interchangeably here, because the distinction is not very important IMO for the purpose of our discussion. Basically, i mean everybody who tries to beat the market using day trading, arbitrage, algo trading, stock picking, etc, etc.
Bonus material: a modern example of a "shoeshine boy who gives stock tips". Well, actually a millennial girl who travels the world and trades stocks online, including triple leveraged ETFs which she doesn't seem to understand very well, e.g. that they are not suitable for long term investing due to decay. Doing pretty well so far, she beat the S&P 500 last year by a few percentage points. What's going to happen when the market declines is a different question, though. Just watch the youtube video for lolz.
https://youtu.be/vk8v3hOT5iY (https://youtu.be/vk8v3hOT5iY)

Well, beating the stock market once is not a proof for anything. She is not really telling anyone her strategy to pick stocks or determine when she sees a local high or low, so we can't really evaluate her strategy.
If she is just blindly picking volatile stocks, she still could beat the market forever. It can happen that you toss a coin 50 times and head is always up.
But still. She beat the sap 500 by 6 points. It would be necessary to have 500,000 invested to make it feasible to invest so much time.

So, your chance of consistently beating the market over a 30 year period is: 0.2^(30)
That's a small number...it's why basically no one has done it.
That's the chance of beating the market 30 years out of 30. Indeed, that's a small number.
That's not required to "beat the market over a 30 year period", though. What matters is ending up with more money in your account after 30 years, not whether you went "undefeated". Beating the market by 2 points 15 years out of 30 and giving up 1 point 15 years is a winner.

So, your chance of consistently beating the market over a 30 year period is: 0.2^(30)
That's a small number...it's why basically no one has done it.
That's the chance of beating the market 30 years out of 30. Indeed, that's a small number.
That's not required to "beat the market over a 30 year period", though. What matters is ending up with more money in your account after 30 years, not whether you went "undefeated". Beating the market by 2 points 15 years out of 30 and giving up 1 point 15 years is a winner.
You are right that it's beat the market every year.
I debated whether or not to put the equation in there. I hedged by putting "consistently" in these.
It would be better to add in a bunch more cases where you beat the market...maybe up to the point where you're beating the market ~ 60% of the time (I guess we'd need a definition of what consistently would be). It's still going to be a very small number...especially when the odds that you beat the market at 20%.

So, your chance of consistently beating the market over a 30 year period is: 0.2^(30)
That's a small number...it's why basically no one has done it.
That's the chance of beating the market 30 years out of 30. Indeed, that's a small number.
That's not required to "beat the market over a 30 year period", though. What matters is ending up with more money in your account after 30 years, not whether you went "undefeated". Beating the market by 2 points 15 years out of 30 and giving up 1 point 15 years is a winner.
You are right that it's beat the market every year.
I debated whether or not to put the equation in there. I hedged by putting "consistently" in these.
It would be better to add in a bunch more cases where you beat the market...maybe up to the point where you're beating the market ~ 60% of the time (I guess we'd need a definition of what consistently would be). It's still going to be a very small number...especially when the odds that you beat the market at 20%.
If the chance to beat the market is 50% and you have 10,000,000 trader trying it, after 20 years, estimated 9 traders will have beaten the market every single year.

If someone hasn't already mentioned, the reality of total stock indexes outperforming managed funds is only over a long enough period of time. Per any of Bogel's books on the subject they present ample evidence that over 10, 20 and 30 year investment periods the odds of you outperforming an index drop off sharpely. In the short term there is likely little to no difference between investments with similar goals besides cost. By the end of most 30 year runs the compounding effect of cost means indexes will be ahead of 90% or so of equivalent overpriced funds. If you can find a managed fund that costs the same as an index this would possibly not be true. This is all given in part to the fact that no one has difinitively proved they have a better stick portfolio strategy than buying the total stock market with a basic weighted strategy for all time. If they had indexes would be based on that strategy by now.
The primary driver for this is simply the compounding impact of cost. The higher cost means managed funds are constantly working at a deficit.
If you are trying to compare to a buy and hold strategy involving only managment by the investor I think it is even more complex. Then you run the risk of having a bad strategy, or failing to consider the full cost of trading, selling and buying your individual stocks. Odds are limiting your exposed to less of the market increase your risk to sector bubbles or losing out on gains due to lack of holding in small and mid cap stocks. I don't know of anyone building index like portfolios from individual stocks. Most probably have a bunch of blue chip stocks which are likely systemically hurting their yearly gains with erratic or no growth.

If the chance to beat the market is 50% and you have 10,000,000 trader trying it, after 20 years, estimated 9 traders will have beaten the market every single year.
And your odds of beating the market are not as good as 50/50.

If the chance to beat the market is 50% and you have 10,000,000 trader trying it, after 20 years, estimated 9 traders will have beaten the market every single year.
And your odds of beating the market are not as good as 50/50.
January 1st, you throw a coin. Head: you buy a put, otherwise a call.

Your example is good: you have a 50:50 chance to beat the market. So will it be financially wise to try to beat the market? The expected value of your bet is the return of the market, but you have to invest time and fees, so the real expected value is lower.
No you do not have a 50% chance of beating the market in any given year. That would be true if half of the stocks returned more than the index and half returned less, but it doesn't necessarily work that way.
Suppose the index were composed of three identical portions of stocks, and the entire index gained 10%. Two of the stocks gained 5%, while one stock gained 20%.
If you are picking a single stock at random, you only have a 1/3 chance of beating the index.

January 1st, you throw a coin. Head: you buy a put, otherwise a call.
Presumably there are costs associated with these transactions.

If the chance to beat the market is 50% and you have 10,000,000 trader trying it, after 20 years, estimated 9 traders will have beaten the market every single year.
And your odds of beating the market are not as good as 50/50.
January 1st, you throw a coin. Head: you buy a put, otherwise a call.
Depends, is the question asking for a set of boolean's as in "beats the market vs does not beet the market during each X years" or a final compounded dollar value from "accumulated return over X years". The former does not account for gains and losses not being equal; you may beat the market 9 of 10 years and still end up with fewer dollars at the end vs buy/hold. Beating the market 9 of 10 years might get you bragging rights but it may not buy a cup of coffee.

Depends, is the question asking for a set of boolean's as in "beats the market vs does not beet the market during each X years" or a final compounded dollar value from "accumulated return over X years". The former does not account for gains and losses not being equal; you may beat the market 9 of 10 years and still end up with fewer dollars at the end vs buy/hold. Beating the market 9 of 10 years might get you bragging rights but it may not buy a cup of coffee.
I would only care about this question if the test for win/lose was ending up with more money over a period of 30yrs since that is the timeframe I care about for FIRE.

Seadog,
You're only 33 years old. You've only known the greatest bull market your entire investing life. Take time into account for your theories.
Pffst! Millennials! Lol.... kidding
Sort of related to this point: Most people retired now haven't actually had that bad an outcome with their investing.
Even people who retired end of 2008 didn't, when you consider the two decades before that, end up with that bad a return.
The Y2k retiree is having a rough time.

Depends, is the question asking for a set of boolean's as in "beats the market vs does not beet the market during each X years" or a final compounded dollar value from "accumulated return over X years". The former does not account for gains and losses not being equal; you may beat the market 9 of 10 years and still end up with fewer dollars at the end vs buy/hold. Beating the market 9 of 10 years might get you bragging rights but it may not buy a cup of coffee.
I would only care about this question if the test for win/lose was ending up with more money over a period of 30yrs since that is the timeframe I care about for FIRE.
And it's probably better to look at a longer time horizon (i.e. estimate for your lifespan  current age) because returns matter when you are accumulating and when you're spending down. The longer the time horizon, the better the index will do.

The longer the time horizon, the better the index will do.
What does this mean? If it was 2009 you wouldn't think that additional time had helped the index any at all.

The longer the time horizon, the better the index will do.
What does this mean? If it was 2009 you wouldn't think that additional time had helped the index any at all.
Eight years isn't that long.
If someone hasn't already mentioned, the reality of total stock indexes outperforming managed funds is only over a long enough period of time. Per any of Bogel's books on the subject they present ample evidence that over 10, 20 and 30 year investment periods the odds of you outperforming an index drop off sharpely.

The longer the time horizon, the better the index will do.
What does this mean? If it was 2009 you wouldn't think that additional time had helped the index any at all.
The problem isn't beating the market, it's beating the market consistently. My understanding is that the data show that people who beat the market next year isn't well correlated to people who beat the market last year.
So, you're rolling a 10 sided die and there are only 2 numbers you win on (you lose on 8). Lets say the sides are numbered 110 and you win on 9 and 10.
If you throw three times, you might get a lucky run and roll an 9 or a 10 on all of your throws. If you threw a much larger number of times, you'd expect that you'd be much closer to 20% wins and 80% losses. Randomness matters more in smaller sample sizes and less in large sample sizes.
When you buy an index fund you know that you will get the market performance minus the fee. You guarantee that you will not beat the market, but your underperformance is something like 0.05%.
When you pick stocks or go with an actively managed fund, you throw the dice. And because the house takes a cut, the odds are weighted against you. The more times you throw, the more likely you are to end up with the expected value (which is you underperforming the market). It is not impossible that you will beat the market, but it's very unlikely. It is also likely that you will underperform the no effort option (index funds).
Furthermore, a strategy that is more active is probably more likely to have negative effects due to behavior. If you believe that you can do things like time the market, you're more likely to try to time the market during volatile time periods. I'm pretty sure that the statistics for market timing are worse than those for buy and hold.

The longer the time horizon, the better the index will do.
What does this mean? If it was 2009 you wouldn't think that additional time had helped the index any at all.
I'm not talking about an 8yr time horizon. My point is that a longerterm time horizon does not guarantee higher returns, which is the only translation of "better" that I can imagine. maybe what you meant is, the longer the time horizon, the greater probability that an index fund will beat some other strategy?

No you do not have a 50% chance of beating the market in any given year. That would be true if half of the stocks returned more than the index and half returned less, but it doesn't necessarily work that way.
Suppose the index were composed of three identical portions of stocks, and the entire index gained 10%. Two of the stocks gained 5%, while one stock gained 20%.
If you are picking a single stock at random, you only have a 1/3 chance of beating the index.
In that situation, if you want to maximize your singleyear chances of beating the market, you should buy 50% of two of the stocks, which will give you a 2/3 chance of beating the market. (Basically, you're picking the one stock to NOT invest in.)

OK, I can see I was sort of right and wrong. I was more curious about picking a few specific stocks then passively holding, not actively managed by someone else buying and selling lots which I agree hand down should be worse. It came up from somebody lambasting someone else in a different thread because they only had a dozen or so stocks vs indexed ETFs.
And it's probably better to look at a longer time horizon (i.e. estimate for your lifespan  current age) because returns matter when you are accumulating and when you're spending down. The longer the time horizon, the better the index will do.
In this sense, and what's considered long term? like the theory of large numbers/dice example, you'd expect an average of 3.5, but only after a sufficiently large number of rolls. If you pick 10% of the index, or 25% or 90%, how many "rolls" (or years?) would you have to wait to get to your expected return of market returns?
Similarly to the lottery. Your expected value of that $1 lotto tickets is probably 75 cents "long term", however because it would need to play out over a hundred million draws to realize that effective value, it's somewhat a moot point.
Therein lay one of the problems with markets. Just how much timing (and as a consequence  luck) has to do with it. Retiring in 2001, 2009 or just after any huge decline would be night and day compared to a couple years before. Numerically you could say "just wait, a few more rolls of the dice, it will revert to the mean", which is likely true, unfortunately having finite life spans does make it an issue.

"You can't beat the index" has almost become a bit of a religion it seems to some people. Not saying I wholly disagree, but I'd like to know why exactly, backed up with some numbers.
My school of thought is that an individual, or random spattering of stocks bought and held should have a 50% chance of beating the market approaching market returns (decreasing variance) as you trend toward owning every stock in the index. Why wouldn't that be the case?
Then too it follows, that if stock picking truly is a losing game, then it seems to me the inverse would be a winning one. Namely, buying the entire index, *except* for a few "chosen" stocks. Since you can expect those chosen stocks to underperform the market, the remainder that you didn't "choose" would exceed it no? Similarly, play the same choosing game, but just go short since you can confidently say it will do worse than the market.
I simply can't reconcile how 90+% or whatever the quoted number is can underperform. Simply because when you look at a large enough sampling of funds, it would be essentially mimic the entire market no?
The 90% # I often see floated around is based off of taking yearly returns over a specific period of time, so the way the math is done almost guarantees very few will 'win'. If I have 10 money managers and 5 outperform the market on year one, now I continue counting with 5. If 3 of those 5 outperform the market on year 2 as well now we are already at 70% of active managers underperform the market. It's a conditional probability problem.
The funny thing is I don't care about that at all. I'd be fine if my portfolio underperformed the market in any given year as long as my total return over 20 years beats the market.
Having said that I still don't want to fork over 1% for active management.

The 90% # I often see floated around is based off of taking yearly returns over a specific period of time, so the way the math is done almost guarantees very few will 'win'.
My understanding is that it's that only ~20% of actively managed funds beat the market after fees every year. But, I think the source I have for this number is the Stacking Benjamin's Podcast, so take that for whatever it's worth.

Seadog,
You're only 33 years old. You've only known the greatest bull market your entire investing life. Take time into account for your theories.
Pffst! Millennials! Lol.... kidding
Sort of related to this point: Most people retired now haven't actually had that bad an outcome with their investing.
Even people who retired end of 2008 didn't, when you consider the two decades before that, end up with that bad a return.
The Y2k retiree is having a rough time.
Not sure what you mean when you reference the Y2K retiree. I.e., the dot com implosion, the Great Recession, etc. And I agree that all of these things would have stressed that Y2K retiree.
But one other way to look at this is like this: If this person retired after 35 years of retirement savings and used a 75% stocks and 25% bonds asset allocation, he or she ended up with about 50% more in their retirement account than the median outcome. And maybe earned an IRR on their investment that runs about 8% rather than 150 average of 6%ish.
I.e., $5500 a year for 35 years grows to about $620K if asset allocation equals 75% stocks and 25% bonds on average.
But the Y2K retiree ended up with about $950K by my calculation... an amount which reflects about an 8% IRR.

Namely, buying the entire index, *except* for a few "chosen" stocks. Since you can expect those chosen stocks to underperform the market,
Why?
No, any stock in the market has a 50% chance of underperfoming, and 50% chance of outperforming. Doesn't matter what you choose.

No, any stock in the market has a 50% chance of underperfoming, and 50% chance of outperforming. Doesn't matter what you choose.
That's not true. Market returns are driven by a a small subset of high performing stocks. That means most stocks under perform the market. If you are picking randomly you will get losers [compared to the market] far more often than you will get winners.

Namely, buying the entire index, *except* for a few "chosen" stocks. Since you can expect those chosen stocks to underperform the market,
Why?
No, any stock in the market has a 50% chance of underperfoming, and 50% chance of outperforming. Doesn't matter what you choose.
No. 'The market' has weights on each stock. Any given stock will under or over perform  this does not mean the odds are 5050; I will or will not be struck by lightning today but the odds of this are not 5050.
" Since you can expect those chosen stocks to underperform the market," yes. this is stock picking (duh) and the entire point is that stock picking under preforms in the long term for the vast majority of trials.

OK, I can see I was sort of right and wrong. I was more curious about picking a few specific stocks then passively holding, not actively managed by someone else buying and selling lots which I agree hand down should be worse.
The fact that you have picked a few stocks rather than the entire index means you are actively managing as well, and not passive. I don't think it has to do with how often transactions occur.

That's not true. Market returns are driven by a a small subset of high performing stocks. That means most stocks under perform the market. If you are picking randomly you will get losers [compared to the market] far more often than you will get winners.
If you randomly pick from the market using the same weighting as the market itself you will get the same results as the market (assuming sufficient number of trials and/or large numbers of randomly selected stocks etc etc). Basically this recreates the market with a smaller number of stocks.

No. 'The market' has weights on each stock. Any given stock will under or over perform  this does not mean the odds are 5050; I will or will not be struck by lightning today but the odds of this are not 5050.
I think the weight of a stock is confusing the issue. Yes, different stocks have different weights in an index but the important topic is the distribution of individual stock returns. If you put $5,000 on each of ten individual stocks, it doesn't really matter how much weight they have in an index because in your portfolio they each start with 10% weight. The important question is, how will they perform.

No. 'The market' has weights on each stock. Any given stock will under or over perform  this does not mean the odds are 5050; I will or will not be struck by lightning today but the odds of this are not 5050.
I think the weight of a stock is confusing the issue. Yes, different stocks have different weights in an index but the important topic is the distribution of individual stock returns. If you put $5,000 on each of ten individual stocks, it doesn't really matter how much weight they have in an index because in your portfolio they each start with 10% weight. The important question is, how will they perform.
It still matters how you selected your basket of stocks. You have basically made an equally weighted index that you are comparing to a nonequally weighted index. On a practical level running one trial with a small number of stocks your results will be highly variable.
You could run some simulation tests in Excel/GoogleSheets if you really wanted.
Edit:
With equal weighting selection your custom index would be over representative of lower weighted stocks who have smaller market caps so it would resemble a small/mid cap index. So it might on average beet 'the market' but you would have a volatility penalty and you could just have bought NAESX.

You guys are confused because the weights in this case are how the stocks perform.
For example, there are ten equally weighted stocks in the entire market.
One triples in value
One doubles in value.
Three stay the same.
Four lose half their value
One goes bankrupt.
The mean value of the entire market stays the same.
The median value of the market decreases by 25%
The mode value decreases by half.
If you had an index, you would break even.
If you picked one stock, you would beat the market with a 25% chance, break even with a 25% chance, and lose to the index with a 50% chance.
The results are asymmetrical because the probability mass function has a high degree of skewness, meaning the high performing stocks are rare and perform much better than average while the lower performing stocks are more common and only perform a little worse than average.

You guys are confused because the weights in this case are how the stocks perform.
For example, there are ten equally weighted stocks in the entire market.
One triples in value
One doubles in value.
Three stay the same.
Four lose half their value
One goes bankrupt.
The mean value of the entire market stays the same.
The median value of the market decreases by 25%
The mode value decreases by half.
If you had an index, you would break even.
If you picked one stock, you would beat the market with a 25% chance, break even with a 25% chance, and lose to the index with a 50% chance.
The results are asymmetrical because the probability mass function has a high degree of skewness, meaning the high performing stocks are rare and perform much better than average while the lower performing stocks are more common and only perform a little worse than average.
This. I think that you'd expect stock performance to follow a Pareto distribution where ~20% of the stocks provide ~80% of the gains.
You don't know what the 20% is before you start, so just picking random stocks doesn't give you a 50% chance of beating the index.

Here is some discussion around the skewed distribution of returns and the impact of this phenomenon on trying to pick winning stocks vs. buying the index.
https://www.bloomberg.com/news/articles/20170409/lopsidedstocksandthemathexplainingactivemanagerfutility
^This explains the math the OP is looking for very well.

The definitive analysis on this topic was done by Fama and French.
https://famafrench.dimensional.com/essays/luckversusskillinmutualfundperformance.aspx
They looked at historical mutual fund performance (and accounted for survival bias). Main conclusion that, accounting for fees, active management only sometimes does better than the index. But this sometimes is as much as you'd expect by plain luck.

Answer: Efficient Market Hypothesis

I don't buy the EMH, certainly not the strong form. Even so, the "low cost hypothesis" of Bogle (and others) is really compelling. Even if you assumed (healthy) 10% nominal equity returns, giving up 1% plus taxes and trading costs of that is a HUGE portion and hurdle. I'm evolving more into an indexer with a value tilt (or a reduction in the inherent momo tilt in cap weighted indexes).