I think the concern is that, notwithstanding Warren Buffett's amazing track record, you're still betting on someone doing better than the market, which statistically people don't do. Warren Buffett is an outlier, but it's unclear whether he's a statistical outlier (if enough people try something, some are going to be successful), or whether he truly has an analytical advantage.
If he's the former, then there's no reason to believe he's likely to outperform the market in any given future year. And indeed, I think he's (slightly) behind the S&P 500 the last 10 years. If the latter, then you have to factor in how long he's going to be doing it, and whether his analytical advantage over the market is still sufficiently great to make it worth taking a chance on him. Again, the last 10 years suggest that whatever his advantage is may not be as great anymore.
In any event, I think Warren Buffett is an incredibly intelligent guy who says incredibly intelligent things. But I'm skeptical whether he's going to outperform the market in any given year going forward. Thus, index funds.