Did you subtract any tax benefits of the mortgage from the rate?
If you deduct mortgage interest on your taxes, that could push the effective "after tax" rate to 3.x%, and it might be even less appealing to pay it off. Most people are in the 22% income tax bracket, and if that's your situation, you're paying 3.6% on your mortgage after taxes. Right now 30 year treasuries are paying 3.0%, and your mortgage is only 0.6% above that (after tax).
Future returns can't be predicted. In a white paper from Vanguard various metrics were used and the best was CAPE 10 (P/E ratio averaged over 10 years, "cyclically adjusted price/earnings"). CAPE 10 was 0.4 correlated with future returns - not a good prediction.
https://personal.vanguard.com/pdf/s338.pdfYou've been putting $18,000 extra into your mortgage for 5 years, to avoid paying maybe 3.6% (my after tax estimate) in interest. The stock market has gained about 13.3% per year, an almost +10% higher return than the choice you've been making. Would you invest if the alternative cost you $150/month?