Gosh, honestly, I'd be tempted to refinance to a 15-year mortgage and go all-in with your whole 50K so your principal balance is $100K. Your payments would be quite low, so you could rebuild your emergency fund pretty quickly.
How come you didn't go for a 15-year loan to begin with?
If it was because you felt that with interest rates this low, you wanted to lock the rate in for the longest term possible, then following that logic would lead you to invest in the market vs. paying down your mortgage.
These things are so personal, really. Often, it's way more about psychology and your personal comfort level vs. the numbers. And the larger economy does what it does regardless, and it can make a huge difference in where you end up in life financially, too. We like to think we're smart when we do well in the market or in real estate, but we're really just riding the economic wave.
Living well below your means is the most important thing you can do, and you're doing it. Having liquidity really helps in bad financial times, and I just advised against that, didn't I? So don't listen to me, or only do so with a huge grain of salt!