At what % of passive market ownership would you start picking stocks?

[Radagast]

I go with essentially never. I reckon there will always be at least a million people out there who would jump in to exploit inefficiencies ahead of me, and at that point why bother. But if it started to get really inefficient to the point that one or two were obvious to me I might jump in. So far that happened once and I didn't act on it.

It barely takes any transactions for a market to come up with a good price. When I bought my house it was one of only three multifamily residences to have sold within a year and a mile, but extrapolating other house prices and those three it was pretty easy to guess what a good price should be, and that is in addition to fundamental methods of determining price like % cashflow. If two participants agree to a trade, there is a new data point. What is the worst case, we return to the trading volumes of 1928? I think that would be plenty to determine accurate prices, especially with the greater knowledge, transparency, and computing power now available.

[MustacheAndaHalf]

But you agree with the premise that there is *less* efficiency when a greater share of the market is held in index funds, right?

I get that there's still a ton of active $ left to pick stocks even at 80% passive. But the fact that there's less should correlate to less efficiency, right?

First a little demonstration - let's say a fund does nothing with 80% of it's assets.  What is it's annual turnover?

Well it's not 20%, because if the fund turns over that 20% every month, that's 240% turnover.  Or if it crazily turns over every week, that exceeds 1000% turnover per year.  And here's my point: the stock market trades 5 days a week at millisecond speeds.  There's plenty of room for trading in that remaining 20% of assets, because it matters how fast they change hands rather than the $ amount.

An active fund has lots of people working for it: the portfolio manager, researchers, traders.  I claim some funds aren't finding anything before the market does.  They are not providing any additional return.  Maybe investors liked their marketing, and are still doing better than a savings account, or are ignoring their accounts.  Active funds can persist even when they don't find anything valuable in the stock market.  I claim there are a lot of these people, and the market is saturated with people trying to exploit news and opportunities.  Losing some of these people doesn't reduce market efficiency - it actually just weeds out weak competition.

The real inefficiency of mutual funds, in my opinion, is all the people who are poorly informed.  They allow inefficient mutual funds to stay around.  I think the market is saturated with them, and losing a lot of them won't impact efficiency at all: because they aren't helping the market remain efficient.

Take SPIVA's data on how often large-cap active funds beat the S&P 500.  They haven't put out full 2018 data yet, so here's the mid-2018 data for active vs S&P 500:
past 1 year, S&P 500 beats 63.46% of active funds
past 3 years, S&P 500 beats 78.64% of active funds
past 5 years, S&P 500 beats 76.49% of active funds
past 10 years, S&P 500 beats 89.15% of active funds
past 15 years, S&P 500 beats 92.43% of active funds
https://us.spindices.com/documents/spiva/spiva-us-mid-year-2018.pdf

If you're worried about a crash, the 10 year data includes the aftermath of the 2008 crisis, and the 15 year performance includes the 2008 crisis.

We're not in the same financial world anymore, so I'm not willing to take these platitudes for granted anymore.

If you don't take the time to think about why your strategy works now and why it will work in the future, you don't have a good plan.

Several books I've read say "This time it's different" are the most dangerous words in investing.

[Classical_Liberal]

I didn't suss out that you might be thinking of precious metals, but if that's what you were thinking of, my answer is no.

I wasn't necessarily. More so I meant breaking your index investing into differing types of stocks, specialized indexes.  ie some that preform well in less than perfect economic situations, certain international, etc. There are classes of equities that move very differently from one another.

[Classical_Liberal]

The PM allocation would reduce volatility, but so what? By wanting that lower volatility, IMO you've created the requirement that your portfolio be 10% larger. So for a retiree who needs a $40k income, your minimum required FIRE portfolio just grew from $1,000,000 to $1,100,000. Would you be willing to sacrifice an extra two years of your life earning that extra $100k just to have coins in a safe or GLD in your brokerage account? Just to reduce the size of the zig zags you see on Mint.com's net worth graph? I'll take my chances but YMMV.

Forget about a gold or PM's for a moment.  That's not a discussion I want to have on these boards.  Your underlying assumption here is just wrong.  You do not need a bigger portfolio with reduced returns IF the standard deviation of returns is reduced along with the ROI. There is research to back this up, I'm just don't have it in from of me at the moment and Google isn't helping, I will post at a later date.  It's also common sense though, if you think about it.  Since the 4% rule is a near worst case, it is based on outliers anyway.  If you can eliminate those outliers your WR can come up. 

I would argue the best time to use a decreased ROI/decreased volatility option is when someone is at the highest risk for Sequence of returns risk and/or when the measure of the higher return portion of your portfolio  shows a potential for poorer than normal returns... So the first decade or so of drawdowns and/or in high CAPE situations.  If the higher volatility/return portion of the portfolio does well, it doesn't matter anyway, you just have less "too much" than you would have in 100% VTI.  OTOH, if it does very poorly (outlier), the noncorrelated portion of the portfolio saves your ass and significantly increases WR.

[soccerluvof4]

I just don't feel this will ever become an issue because there will always be plenty of people that think they can beat the system, young people just starting to invest that haven't heard about indexing and so on. I don't know about most people here but seems like most people I talk to use someone to do there investing. Which is one of the reasons I am here. When i just tell them to do it themselves I always get the same response. " yea but my guy is really good" ! So because of greed as well I just dont see Passive investing getting to a point that would change the Landscape enough.

[FI-King_Awesome]

Thanks for sparking this debate - interesting arguments from both sides.

Personally, I am far too heavily weighted in handpicked stocks. In fact, besides my target date retirement funds, I only carry stocks. No indexes.

It started with stock options, an ESPP, picked up a competitor and put some money into Apple just for fun. The portfolio has done tremendously well, but I doubt it will continue forever. That said, I prefer to invest in something I know - my current company’s industry is something I understand fairly well...

Personally, I’m just getting started with reviewing indexes and am looking for something targeting companies that will benefit from the aging population, increasing middle class, increasing life style diseases, advancements in personalized medicine, etc. I.e. companies in the medical device, consumables industry. I’ve handpicked a few stocks that have done well, but would like to diversify across a larger pool of companies to protect myself against a dog going tits up from poor management.

What about shorts?  I’d like to learn how I can short the entire millennial non-industry of “influencers”...

[nereo]

What about shorts?  I’d like to learn how I can short the entire millennial non-industry of “influencers”...

Not sure what you mean by this, unless you are just hating on a particular generation.  The so-called "influencers' are paid by larger corporations as non-traditional part of their advertising budget - so you'd really need to short these larger companies.  Only I don't see how that would work, as they spend money on these 'influencers' because they believe it lifts the profile of their particular brand, and from an  advertising standpoint its really dirt cheap - often its just some product in exchange for ads and favorable mentions.

[MaaS]

I expect the indexing growth rate to slow down considerably and perhaps even temporarily reverse.

It's easy to dump money into an index when it generally goes up. Not so when it's generally going down.  My guess is that the next real downturn lessens the index enthusiasm for awhile.  Perhaps not on this board, but within the broader population.

So to answer the question: I have no idea on the polling options, but I don't think this issue is necessarily as looming the charts would lead you to believe.

[J Boogie]

I didn't suss out that you might be thinking of precious metals, but if that's what you were thinking of, my answer is no.

I wasn't necessarily. More so I meant breaking your index investing into differing types of stocks, specialized indexes.  ie some that preform well in less than perfect economic situations, certain international, etc. There are classes of equities that move very differently from one another.

Yeah, for sure. The steadiness of dividend aristocrats inspire confidence. They are generally priced accordingly but some go on sale from time to time. I'll probably keep my core holding of SDY and sell of portions of it when individual dividend aristocrats (that meet certain financial criteria as well as having forward facing strategies) get beat up.

[J Boogie]

Robots?  When did we get robots in this discussion??

I am using robots as shorthand for the programs running on the supercomputers you mentioned that execute the algorithms. I also suspect there are plenty of trading firms testing out/utilizing machine learning or AI developed algorithms.

ok.  That's not what a robot is.  An algorithm ≠ a robot.   An algorithm is just a set of operations that someone wrote. 
Here's a simple example: select companies >= $1B, filter out those with new management, declining cashflow or P/E >the 1.5x mean for that sector.  Sort by increasing net profits.
It's the same whether you do it by hand, via an excel spreadsheet or if you write a script.  Of course utilizing a computer is faster.
Saying that computers are limited compared to the human mind is like saying a hammer is limited compared to the carpenter.  One is a tool - the other is the operator. The carpenter isn't going to toss out the hammer, nor is the active manager going to throw out computers. Saying robots algorithms evaluate metrics nearly perfectly misses the whole point.  A 'smarter human' can write a better algorithm, and s/he will thereby get an advantage.

My statement quite clearly indicates I realize an algorithm is not a robot. A robot is a machine capable of carrying out a complex series of actions automatically. That wikipedia definition jives perfectly with my shorthand use of robot.

The whole reason I thought you were referring to AI is because you did not lend any credence to my argument that algorithms are not able to perform nuanced analysis. You can write an algo that will analyze financials, but not one that assesses forward facing strategy based on statements, acquisitions, etc. Many hedge funds have been recently utilizing AI, and it doesn't seem to give them all that much of an edge.

https://www.bloomberg.com/news/articles/2018-03-12/robot-takeover-stalls-in-worst-slump-for-ai-funds-on-record

Anyways.

I could care less how fast other investors' computers run, because I'm not a day trader. I could care less how well they've written their algorithms, because I analyze more than readily digestible data points. And because of that, I don't find your initial point all that compelling.

That's not to say I'm right, (as others have made interesting and compelling arguments about market efficiency) but your argument about algos and fast computers isn't proving me wrong. I'm not Warren Buffet but theoretically I could be - and your point would fall flat.

[MustacheAndaHalf]

It's easy to dump money into an index when it generally goes up. Not so when it's generally going down.  My guess is that the next real downturn lessens the index enthusiasm for awhile.  Perhaps not on this board, but within the broader population.

Active funds tend to shift to cash after the market crash, and that cash hurts their returns when the recovery kicks in.  Index funds remain fully invested.  If you look at 15 years of SPIVA data on the S&P 500 vs active funds, the S&P 500 beats over 90% of them - a time that included the 2008 crisis.

[nereo]

Robots?  When did we get robots in this discussion??

I am using robots as shorthand for the programs running on the supercomputers you mentioned that execute the algorithms. I also suspect there are plenty of trading firms testing out/utilizing machine learning or AI developed algorithms.

ok.  That's not what a robot is.  An algorithm ≠ a robot.   An algorithm is just a set of operations that someone wrote. 
Here's a simple example: select companies >= $1B, filter out those with new management, declining cashflow or P/E >the 1.5x mean for that sector.  Sort by increasing net profits.
It's the same whether you do it by hand, via an excel spreadsheet or if you write a script.  Of course utilizing a computer is faster.
Saying that computers are limited compared to the human mind is like saying a hammer is limited compared to the carpenter.  One is a tool - the other is the operator. The carpenter isn't going to toss out the hammer, nor is the active manager going to throw out computers. Saying robots algorithms evaluate metrics nearly perfectly misses the whole point.  A 'smarter human' can write a better algorithm, and s/he will thereby get an advantage.

My statement quite clearly indicates I realize an algorithm is not a robot. A robot is a machine capable of carrying out a complex series of actions automatically. That wikipedia definition jives perfectly with my shorthand use of robot.

The whole reason I thought you were referring to AI is because you did not lend any credence to my argument that algorithms are not able to perform nuanced analysis. You can write an algo that will analyze financials, but not one that assesses forward facing strategy based on statements, acquisitions, etc. Many hedge funds have been recently utilizing AI, and it doesn't seem to give them all that much of an edge.

https://www.bloomberg.com/news/articles/2018-03-12/robot-takeover-stalls-in-worst-slump-for-ai-funds-on-record

Anyways.

I could care less how fast other investors' computers run, because I'm not a day trader. I could care less how well they've written their algorithms, because I analyze more than readily digestible data points. And because of that, I don't find your initial point all that compelling.

That's not to say I'm right, (as others have made interesting and compelling arguments about market efficiency) but your argument about algos and fast computers isn't proving me wrong. I'm not Warren Buffet but theoretically I could be - and your point would fall flat.

You're straying way off topic here and overlooking the underlying logic. 

In your OP you've asked at what percentage of market ownership would you start picking stocks, and asserted - repeatedly - that indexing must increase inefficiencies in the market which you as an individual investor could then exploit to 'beat the market'.  In a type of technology word-salad you've evoked algorithms, robots, AI and machine learning, while simultaneously saying - repeatedly - that you could care less about these things because you are not a day trader.

But here's the underlying flaw in that logic.  Regardless of whether you use such methods is immaterial to whether the markets start becoming more inefficient, because brokerages and hedge funds do and have for decades, and they are the ones that can move markets and set prices. Brokerages have been employing computers and algorithms since before index funds existed, and they will continue in the future.  Which means as an individual investor picking stocks you have to find these inefficiencies and exploit them before they do - which seems incredibly unlikely.  Even if you think there's a way to better evaluate stocks which isn't part of previous methods, there are literally tens of thousands of hedge-fund managers trying to do the exact same thing as you, only they have the ability to trade in volumes and frequency that you can't match.  They can take a very small edge and turn it into a big profit, whereas even if you could identify a small edge before they do your small, infrequent trades won't net you very much (if at all) after taxes. As we've discussed there remains so much actively managed money floating around that these advantages are vanishingly small.

[J Boogie]

Probably not. In addition to a decent return during a recession, I am also uncomfortable with the reasons we embrace indexing.

We don't really use any solid logical justification as to why it is better, we tend to use the same logic as the older employees in the companies we work at - We do it this way because this way has always been superior no matter what any fancy new person says. It just works. This type of thinking always holds up until it doesn't. We're not in the same financial world anymore, so I'm not willing to take these platitudes for granted anymore.

If you don't take the time to think about why your strategy works now and why it will work in the future, you don't have a good plan. I'm trying to come up with a plan that logically rather than historically has a very good chance at 6%+ annual returns, and hopefully be able to achieve limited loss during recessions.

My issue with index investing as a philosophy is that assumes too many things. I don't completely disagree with the assumptions that the stock market always goes up and that it can be very risky to pick individual stocks, but I certainly don't hold them as gospel and as a thought experiment I want to come up with some conditions that could falsify these assumptions.

I feel like you really need to sit down and read JLCollins' Stock Series.

The reasons you think everyone suggests investing in index funds are addressed by him in a better way than I ever could.

I read it, I thought it was very good.

He mentions the big, ugly event in 1929 and has this to say: "In 2008 we came right to the edge of the abyss.  Closer I think than most folks fully appreciate.  But we didn’t tumble over.  This I find encouraging."

Fed target rates were at 5.25% before the 2008 crash, and now they're at about 2.4%. We've just started shrinking our 4 trillion dollar balance sheet. I'm not sure the Fed has enough flexibility to deal with another recession very effectively in the near future.

I think it is worthwhile to challenge assumptions like "the stock market always goes up" and "you can't beat the market" though I should mention I acknowledge these aren't necessarily related. Getting out of the market and getting into individual stocks are different moves.

[nereo]

I think it is worthwhile to challenge assumptions like "the stock market always goes up" and "you can't beat the market" though I should mention I acknowledge these aren't necessarily related. Getting out of the market and getting into individual stocks are different moves.

These assumptions have been debated at length, and there continues to be an ongoing discussion.  Rather than saying "we should challenge assumptions" it might be more productive to give your thoughts and proposed course of action on each assumption.  With regards to a potential recession and the Fed's reduced capacity to damped its effects, what strategies would you employ?  You've mentioned 'dividend aristocrats' - though it was unclear whether that was a specific strategy or simply an aside about individual stock picking.

[J Boogie]

I think it is worthwhile to challenge assumptions like "the stock market always goes up" and "you can't beat the market" though I should mention I acknowledge these aren't necessarily related. Getting out of the market and getting into individual stocks are different moves.

These assumptions have been debated at length, and there continues to be an ongoing discussion.  Rather than saying "we should challenge assumptions" it might be more productive to give your thoughts and proposed course of action on each assumption.  With regards to a potential recession and the Fed's reduced capacity to damped its effects, what strategies would you employ?  You've mentioned 'dividend aristocrats' - though it was unclear whether that was a specific strategy or simply an aside about individual stock picking.

Fair enough.

I view the dividend aristocrats positively for a few reasons.

Even if the stock market doesn't go up, they've demonstrated a commitment and ability to offer their investors consistent returns. It reduces the speculative nature of the stock market. You can't get rid of speculation 100% as it's always priced in, but their positive momentum inspires confidence. They are what I think of when I hear the phrase "Why work for your money when your money can work for you."

I view picking as a good way to weed out dividend aristocrats whose future profits might be in jeopardy by growing trends such as electric vehicle adoption, cord cutting, etc - and also favor the aristocrats such as Visa who stand to benefit from growing trends like going cashless.

There has been data suggesting dividend aristocrats beat the market, especially during recessions.

I must admit, buying 15 or so of these stocks on slight dips and holding for decades doesn't really correlate all that well to "exploiting inefficiencies" in the market - perhaps that's why this discussion has been a little frustrating for us.

To restate it a bit, I believe that while index fund investing might be "self-cleansing" as JLCollins reminds us, it still throws good money after bad into the dogs that populate the bottom tier of their portfolios until they eventually drop out. This is the inefficiency that I am referring to; and while it can and probably is hastened by aggressive short positions, it still happens for a period of time.

Regarding the Fed's reduced capacity to dampen the effects of a recession and how it would affect my investing... Well, I don't have anything especially insightful to share. I would make sure my portfolio favors needs over wants. IE Kimberly Clark instead of Apple. Not that I've chosen KMB, but you get the idea.

[MaaS]

It's easy to dump money into an index when it generally goes up. Not so when it's generally going down.  My guess is that the next real downturn lessens the index enthusiasm for awhile.  Perhaps not on this board, but within the broader population.

Active funds tend to shift to cash after the market crash, and that cash hurts their returns when the recovery kicks in.  Index funds remain fully invested.  If you look at 15 years of SPIVA data on the S&P 500 vs active funds, the S&P 500 beats over 90% of them - a time that included the 2008 crisis.

Logically, I agree, but the average person isn't going to look at 15 years of SPIVA data and make a completely rational, long-term decision. Us humans tend to be rather illogical. 

My bet is a heck of a lot of people will go back to stock picking once their index fund plunges for a few years straight. We shall see!

[Blueberries]

My statement quite clearly indicates I realize an algorithm is not a robot. A robot is a machine capable of carrying out a complex series of actions automatically. That wikipedia definition jives perfectly with my shorthand use of robot.

The whole reason I thought you were referring to AI is because you did not lend any credence to my argument that algorithms are not able to perform nuanced analysis. You can write an algo that will analyze financials, but not one that assesses forward facing strategy based on statements, acquisitions, etc. Many hedge funds have been recently utilizing AI, and it doesn't seem to give them all that much of an edge.

https://www.bloomberg.com/news/articles/2018-03-12/robot-takeover-stalls-in-worst-slump-for-ai-funds-on-record

Anyways.

I could care less how fast other investors' computers run, because I'm not a day trader. I could care less how well they've written their algorithms, because I analyze more than readily digestible data points. And because of that, I don't find your initial point all that compelling.

That's not to say I'm right, (as others have made interesting and compelling arguments about market efficiency) but your argument about algos and fast computers isn't proving me wrong. I'm not Warren Buffet but theoretically I could be - and your point would fall flat.

You're straying way off topic here and overlooking the underlying logic. 

In your OP you've asked at what percentage of market ownership would you start picking stocks, and asserted - repeatedly - that indexing must increase inefficiencies in the market which you as an individual investor could then exploit to 'beat the market'.  In a type of technology word-salad you've evoked algorithms, robots, AI and machine learning, while simultaneously saying - repeatedly - that you could care less about these things because you are not a day trader.

But here's the underlying flaw in that logic.  Regardless of whether you use such methods is immaterial to whether the markets start becoming more inefficient, because brokerages and hedge funds do and have for decades, and they are the ones that can move markets and set prices. Brokerages have been employing computers and algorithms since before index funds existed, and they will continue in the future.  Which means as an individual investor picking stocks you have to find these inefficiencies and exploit them before they do - which seems incredibly unlikely.  Even if you think there's a way to better evaluate stocks which isn't part of previous methods, there are literally tens of thousands of hedge-fund managers trying to do the exact same thing as you, only they have the ability to trade in volumes and frequency that you can't match.  They can take a very small edge and turn it into a big profit, whereas even if you could identify a small edge before they do your small, infrequent trades won't net you very much (if at all) after taxes. As we've discussed there remains so much actively managed money floating around that these advantages are vanishingly small.

I'm not trying to start a separate argument, but some of your thoughts here are incorrect.  Peter Lynch said it best, "The amateur investor has many built in advantages that could result in outperforming the experts.  Rule #1 is to stop listening to the professionals." 

One of an institution's biggest struggles is what you seem to view as an asset - size.  Concentration is important and when you're handling a $5B fund, it's not easy.  Even tougher with a $250B fund.  Sure, they can move the market and even buy or sell shares with the express purpose of doing the opposite (selling or buying), but it's a risk because they don't know who they're working against.  An individual's ability to move in and out of the market on a $5M or $1M or $50,000 account is vastly different and in respect to size, an advantage. 

[Travis]

My statement quite clearly indicates I realize an algorithm is not a robot. A robot is a machine capable of carrying out a complex series of actions automatically. That wikipedia definition jives perfectly with my shorthand use of robot.

The whole reason I thought you were referring to AI is because you did not lend any credence to my argument that algorithms are not able to perform nuanced analysis. You can write an algo that will analyze financials, but not one that assesses forward facing strategy based on statements, acquisitions, etc. Many hedge funds have been recently utilizing AI, and it doesn't seem to give them all that much of an edge.

https://www.bloomberg.com/news/articles/2018-03-12/robot-takeover-stalls-in-worst-slump-for-ai-funds-on-record

Anyways.

I could care less how fast other investors' computers run, because I'm not a day trader. I could care less how well they've written their algorithms, because I analyze more than readily digestible data points. And because of that, I don't find your initial point all that compelling.

That's not to say I'm right, (as others have made interesting and compelling arguments about market efficiency) but your argument about algos and fast computers isn't proving me wrong. I'm not Warren Buffet but theoretically I could be - and your point would fall flat.

You're straying way off topic here and overlooking the underlying logic. 

In your OP you've asked at what percentage of market ownership would you start picking stocks, and asserted - repeatedly - that indexing must increase inefficiencies in the market which you as an individual investor could then exploit to 'beat the market'.  In a type of technology word-salad you've evoked algorithms, robots, AI and machine learning, while simultaneously saying - repeatedly - that you could care less about these things because you are not a day trader.

But here's the underlying flaw in that logic.  Regardless of whether you use such methods is immaterial to whether the markets start becoming more inefficient, because brokerages and hedge funds do and have for decades, and they are the ones that can move markets and set prices. Brokerages have been employing computers and algorithms since before index funds existed, and they will continue in the future.  Which means as an individual investor picking stocks you have to find these inefficiencies and exploit them before they do - which seems incredibly unlikely.  Even if you think there's a way to better evaluate stocks which isn't part of previous methods, there are literally tens of thousands of hedge-fund managers trying to do the exact same thing as you, only they have the ability to trade in volumes and frequency that you can't match.  They can take a very small edge and turn it into a big profit, whereas even if you could identify a small edge before they do your small, infrequent trades won't net you very much (if at all) after taxes. As we've discussed there remains so much actively managed money floating around that these advantages are vanishingly small.

I'm not trying to start a separate argument, but some of your thoughts here are incorrect.  Peter Lynch said it best, "The amateur investor has many built in advantages that could result in outperforming the experts.  Rule #1 is to stop listening to the professionals." 

One of an institution's biggest struggles is what you seem to view as an asset - size.  Concentration is important and when you're handling a $5B fund, it's not easy.  Even tougher with a $250B fund.  Sure, they can move the market and even buy or sell shares with the express purpose of doing the opposite (selling or buying), but it's a risk because they don't know who they're working against.  An individual's ability to move in and out of the market on a $5M or $1M or $50,000 account is vastly different and in respect to size, an advantage.

Some of the biggest names in the business from the last 50 years had astronomical returns when they were only handling a little bit of money. After they became famous, popular, and ended up with massive amounts of money to manage their performance flattened out because instead of expertly gaming the market, they had so much money to move that they became the market.

[runbikerun]

On efficiency:

There's an assumption on the OP's part that an increase in index holdings means an increase in market inefficiency. Baked into that assumption is a pair of further assumptions: that a larger market is inherently more efficient than a smaller one, independent of absolute size, and that efficiency gains are linear. I'm not convinced that either is true. Maybe all you need for maximum efficiency is for 1% of the money in the market to be liquid and actively traded, and everything past that just adds noise. Maybe the market goes from 99.9 to 99.99% efficiency when you go from 5% to 50% of the market being actively traded, and drops to 99% when only 0.5% of the market is actively managed.

On the idea of the smart analyst with a small stake being more nimble than huge institutional investors: even if you buy into this idea, it doesn't mean you're the nimble brainiac making bank. It's more likely to be a freakishly smart quant at a small hedge fund. I know people who work in those kinds of roles, and they are phenomenally intelligent, work ferociously long hours, specialise in knowing absolutely everything about their particular field, and have nearly unlimited computing firepower. I know enough to know that I will not beat them at active investment, which means they will take my money. Even if there is room in a heavily passive market for active investors to make bank, you and I will not be those active investors.

[ChpBstrd]

The PM allocation would reduce volatility, but so what? By wanting that lower volatility, IMO you've created the requirement that your portfolio be 10% larger. So for a retiree who needs a $40k income, your minimum required FIRE portfolio just grew from $1,000,000 to $1,100,000. Would you be willing to sacrifice an extra two years of your life earning that extra $100k just to have coins in a safe or GLD in your brokerage account? Just to reduce the size of the zig zags you see on Mint.com's net worth graph? I'll take my chances but YMMV.

Forget about a gold or PM's for a moment.  That's not a discussion I want to have on these boards.  Your underlying assumption here is just wrong.  You do not need a bigger portfolio with reduced returns IF the standard deviation of returns is reduced along with the ROI. There is research to back this up, I'm just don't have it in from of me at the moment and Google isn't helping, I will post at a later date.  It's also common sense though, if you think about it.  Since the 4% rule is a near worst case, it is based on outliers anyway.  If you can eliminate those outliers your WR can come up. 

I would argue the best time to use a decreased ROI/decreased volatility option is when someone is at the highest risk for Sequence of returns risk and/or when the measure of the higher return portion of your portfolio  shows a potential for poorer than normal returns... So the first decade or so of drawdowns and/or in high CAPE situations.  If the higher volatility/return portion of the portfolio does well, it doesn't matter anyway, you just have less "too much" than you would have in 100% VTI.  OTOH, if it does very poorly (outlier), the noncorrelated portion of the portfolio saves your ass and significantly increases WR.

So you're saying a higher WR such as 4.5% might be sustainable if one was able to reduce portfolio volatility with a bit of low-correlation, low-yielding ballast? Isn't this concept related to the various optimal asset allocation studies that recommended a 90/10 or 80/20 stock to bond ratio? I believe earlyretirementnow.com has a good probability of failure heatmap for various stock/bond allocations and WRs. That site also recommends an equity glidepath during those first few vulnerable years of retirement.

There is a risk in saying "Instead of 80/20 stocks/bonds, I'll try 80/20 stocks/cash or stocks/PMs or stocks/bitcoin or stocks/uncorrelatedwhatever". The risk is that the AA studies were done on bonds, which yield something, and many low correlation assets yield little or nothing. If the substitution reduces (or has a chance of reducing) the portfolio's earnings potential, the portfolio's odds of long-term survival will be affected - probably for the worse.

An all-cash portfolio with an ROI of zero would be utterly uncorrelated with the stock market, but it would be nearly certain to be depleted in about 20 years at a 4% WR (not 25y because don't forget inflation). Low volatility does not equal safe.

[Classical_Liberal]

So you're saying a higher WR such as 4.5% might be sustainable if one was able to reduce portfolio volatility with a bit of low-correlation, low-yielding ballast? Isn't this concept related to the various optimal asset allocation studies that recommended a 90/10 or 80/20 stock to bond ratio?

Correct.  But I would argue a more specific portfolio of specialized sector index funds, international equities, alternative assets, and specialized bond funds could reduce volatility much more than varying percentages of total market/total bond.  This is the tie in with the OP

I believe earlyretirementnow.com has a good probability of failure heatmap for various stock/bond allocations and WRs. That site also recommends an equity glidepath during those first few vulnerable years of retirement.

I read through that series awhile ago and don't remember exactly which data is presented, I'll take your word for it.  However, yes, this is exactly the purpose of a bond tent.  Reduce volatility at the highest risk points.  Again, this is just a total/total idea though.

There is a risk in saying "Instead of 80/20 stocks/bonds, I'll try 80/20 stocks/cash or stocks/PMs or stocks/bitcoin or stocks/uncorrelatedwhatever". The risk is that the AA studies were done on bonds, which yield something, and many low correlation assets yield little or nothing. If the substitution reduces (or has a chance of reducing) the portfolio's earnings potential, the portfolio's odds of long-term survival will be affected - probably for the worse.

 
Bonds have the potential for negative return too.  This is exactly why the mid 1960's sucked as historical SWR points.  The 1970's brought both shitty equity and bond performance, at least from a total/total standpoint.

An all-cash portfolio with an ROI of zero would be utterly uncorrelated with the stock market, but it would be nearly certain to be depleted in about 20 years at a 4% WR (not 25y because don't forget inflation). Low volatility does not equal safe.

Right, but you only need about 1.25% real to make it last 30 years, 3.25 gets you 50, and obviously 4% is perpetual. Assuming zero volatility in returns, obviously this is the extreme.

The point is you do not need the 7% real equities produce historically. You only need half that, and that 7% average is useless if the first 10 years are negative, too much principle depleted.  So if a person is at significant risk (via high CAPE or early in withdrawal) and assuming the goal is not to run out of money (as opposed to best chances of having too much money), then trading return for reduced volatility makes sense.  It maximizes minimum SWR.

Here's an article wrt the research showing this relationship as previously promised.

[fattest_foot]

I read it, I thought it was very good.

He mentions the big, ugly event in 1929 and has this to say: "In 2008 we came right to the edge of the abyss.  Closer I think than most folks fully appreciate.  But we didn’t tumble over.  This I find encouraging."

Fed target rates were at 5.25% before the 2008 crash, and now they're at about 2.4%. We've just started shrinking our 4 trillion dollar balance sheet. I'm not sure the Fed has enough flexibility to deal with another recession very effectively in the near future.

I think it is worthwhile to challenge assumptions like "the stock market always goes up" and "you can't beat the market" though I should mention I acknowledge these aren't necessarily related. Getting out of the market and getting into individual stocks are different moves.

It's interesting to me that everyone just assumes that the Fed knows what it's doing and that cutting rates during the Great Recession was a positive. We don't actually know because we only have the one timeline to work with. But what if cutting interest rates and QE all actually made a normal market downturn become the event that made the Great Recession possible?

The real truth about the 2008 financial crisis | Brian S. Wesbury | TEDxCountyLineRoad

I'm not saying this guy is right in his speech. Just that it's something to consider.

[nereo]

I'm not trying to start a separate argument, but some of your thoughts here are incorrect.  Peter Lynch said it best, "The amateur investor has many built in advantages that could result in outperforming the experts.  Rule #1 is to stop listening to the professionals." 

One of an institution's biggest struggles is what you seem to view as an asset - size.  Concentration is important and when you're handling a $5B fund, it's not easy.  Even tougher with a $250B fund.  Sure, they can move the market and even buy or sell shares with the express purpose of doing the opposite (selling or buying), but it's a risk because they don't know who they're working against.  An individual's ability to move in and out of the market on a $5M or $1M or $50,000 account is vastly different and in respect to size, an advantage.

I didn't respond at first because runbikerun gave a pretty good synopsis.  I'm not refuting that individual investors have certain advantages that large funds do not - Buffett loves to highlight how his fund is beholden to the whims of its shareholders and has to produce quarterly reports even though his preferred investment timeframe is 'forever'. 
BUT - the OP's premise is that with an increasing share of the market being held in index funds inefficiences will emerge that were not present at lower levels. That is what I was disagreeing with, in part because large funds can command so much volume and leverage.  Admittedly the conversation took a hard turn into side topics, but the advantages of the individual investor holding for the long term are no different now than they were two decades ago, and prices continue to be set by the large funds that buy stocks in large lots on a frequent basis. Ergo, increased indexing will not change the dynamics.

[J Boogie]

I read it, I thought it was very good.

He mentions the big, ugly event in 1929 and has this to say: "In 2008 we came right to the edge of the abyss.  Closer I think than most folks fully appreciate.  But we didn’t tumble over.  This I find encouraging."

Fed target rates were at 5.25% before the 2008 crash, and now they're at about 2.4%. We've just started shrinking our 4 trillion dollar balance sheet. I'm not sure the Fed has enough flexibility to deal with another recession very effectively in the near future.

I think it is worthwhile to challenge assumptions like "the stock market always goes up" and "you can't beat the market" though I should mention I acknowledge these aren't necessarily related. Getting out of the market and getting into individual stocks are different moves.

It's interesting to me that everyone just assumes that the Fed knows what it's doing and that cutting rates during the Great Recession was a positive. We don't actually know because we only have the one timeline to work with. But what if cutting interest rates and QE all actually made a normal market downturn become the event that made the Great Recession possible?

The real truth about the 2008 financial crisis | Brian S. Wesbury | TEDxCountyLineRoad

I'm not saying this guy is right in his speech. Just that it's something to consider.

That's an interesting argument.

I think the dramatic lowering of interest rates to recover from the dot com crash wasn't the only ingredient. It coincided with the loose loan standards to create a housing bubble and then the loose derivative trading standards popped it in a very destructive way. I think the popularity of adjustable rate mortgages during that time period played a role as well.

Mark to market accounting seems to be the most accurate way to assess value. Historical cost might create more stability in the short term but ultimately bears little resemblance to market value. The way this guy shits on M2M accounting makes me think he owns real estate in CA :)

[Blueberries]

Some of the biggest names in the business from the last 50 years had astronomical returns when they were only handling a little bit of money. After they became famous, popular, and ended up with massive amounts of money to manage their performance flattened out because instead of expertly gaming the market, they had so much money to move that they became the market.

Absolutely.

On efficiency:

On the idea of the smart analyst with a small stake being more nimble than huge institutional investors: even if you buy into this idea, it doesn't mean you're the nimble brainiac making bank. It's more likely to be a freakishly smart quant at a small hedge fund. I know people who work in those kinds of roles, and they are phenomenally intelligent, work ferociously long hours, specialise in knowing absolutely everything about their particular field, and have nearly unlimited computing firepower. I know enough to know that I will not beat them at active investment, which means they will take my money. Even if there is room in a heavily passive market for active investors to make bank, you and I will not be those active investors.

True, just because you have an advantage doesn't mean that you're a nimble brainiac making bank.  But, one must not be a nimble brainiac to make bank either. 

Quants typically make horrible traders; many are amazing analysts, but they are usually horrible traders.  Now, in terms of the analysis, they undoubtedly give their firm an advantage the individual investor does not have.  But, that doesn't equate to being a nimble brainiac making bank, as evidenced by the numbers.