Since sept 2008? That's pretty much the bull market. Look longer term and make sure you factor fund expenses.
mefla said that "2008" was his argument, so I went back exactly seven years to 2008. For the record, 2008 was not a bull market. The S&P was down something like -35% that year. If that's pretty much a bull, I'd hate to see a bear.
But since you don't like 2008, lets go back to Sept. 2007 which was the worst possible time to be in the market, pretty much the peak right before it crashed, and we get....5.6% annualized, including dividends. Remove another 0.02% for expenses, if you like.
However, I agree the time frames are not nearly long enough (looking at returns YTD is a red herring). Since the length of a typical mortgage is 30 years, it only makes sense to look at likely returns over 30-year periods. Going back to Sept. 1985, we get 10.6% annualized, including dividends.
In fact, if we look closer, we find there are no 30 year periods where the S&P500, including dividends, returned less than 8% annualized. The median is 10%, the high 14%.
I've done this calculation from every angle, using a blue million different assumptions, different rates of return, different tax assumptions, etc. And there are no reasonable set of assumptions where paying off the mortgage makes sense. In every scenario, you wind up with moderately to
vastly more money at the end of 30 years if you simply invest. Yes, the future could be worse than past, etc. but that's the math.
Again, I want to emphasize, in your situation you are absolutely correct that you should not be in the stock market. If the volatility keeps you up at night or otherwise makes you feel regret, then do not invest there and pay down the house instead. That goes triply so if you view the market being down YTD as some sort of validation of your views.