Congrats on your great real estate investment!
In your shoes, I would just pay off the house. Sure, you could continue to invest in the market, but it sounds like you already have a hefty stash invested. Paying off my home would bring me peace of mind, and it sounds like you feel the same way. It's also a hedge against market risk, of course. Once it's paid off and you have no mortgage payment, then you'll have EVEN MORE cash per month that you can stash.
As you have already read, there are a fair number of people who think you should leverage yourself to the hilt using your house as collateral so you can invest in other things. Personally, I like having somewhere secure to live. I'd love to know the age of the folks giving this advice. It reeks of a mentality I heard from people nineties, including from a bunch of people who really didn't know anything at all about markets but had dubbed themselves financial geniuses. I just know too many people who have lost their homes to consider this (including 2 neighbors on my tiny cul de sac, both near retirement, the last time the market tanked!)
Whatever you decide, good luck!