It's also important to note that the return you actually experience is mathematically different from the average.
Since 1900, the average inflation-adjusted return of the S&P500 (including reinvested dividends) is 8.5%. The compound annual growth rate that you would have actually experienced if you were fully invested over that timeframe is only 6.6%. Pretty big difference! The difference is a result of
volatility drag. Basically, the more volatile the investment, the more your actual returns will trail the average. You can read about the math at the link.
Even looking at the more rosy averages, there have been many rolling periods where the stock market lost money (inflation adjusted) for more than a decade. Occasionally as long as 20 years! (
PDF) Recently, a 100% stock market investor starting in 2000 experienced 13 years of negative real returns before finally breaking even for good. Stocks absolutely do work out well in the long run, but if you're not prepared to wait them out for at least 10 years without questioning your plan and switching course, you may be in for a tough emotional ride. And remember, if you switch portfolios all the time you generally lose money.
One positive thing about diversification is that you can trade off a little of that average return but gain a much less volatile return in the process. Shorter down periods are a lot easier to weather as an investor, and because of the effect of volatility drag, the difference in CAGRs between a "high risk high return" portfolio and a "lower risk lower return" portfolio is often less than you might think by simply looking at the average returns.
TL;DR: Don't be freaked out about the stock market. But also be realistic about its history. Find a middle ground with a diverse portfolio you can truly stick with through ups and downs, and you'll most likely be a lot happier and wealthier in the long run.