Yes, but with caveats. Small cap vs large cap has worked out favorably and unfavorably over rolling 10 year periods in the past century. Sometimes the behavior has been very pronounced in one direction or the other. This can make a huge difference in sequence risk, but no one knows what will be winning or losing in the years to come.
That said, making the "right" decision would be far from a rounding error in recent history. It would profoundly change the realized returns in a way that could make or break a portfolio. The answer, as usual, is to diversify, own a good bit of everything, accept that your portfolio will never or almost never be the winner running ahead of the pack, and stay the course.
Agree with your larger point, but some context in regards to the original question: Because both indices are cap weighted, both are pretty much a large cap strategy. That's why they track so closely.
@mistymoney The total market index and 500 index are pretty close to the same thing, for the reasons mentioned above. If total market is available in your 401(K) I personally would do that. If only 500 index is available, I'd do that and sleep like a baby. There is no reason to split the difference between the two.
Now, if you really want to go down a rabbit hole you can stir in small and mid-cap index funds, maybe some international etc. And there is nothing wrong with that. That's what I do. But there is a real beauty in just buying the whole market and being done.