Thanks for the replies everyone, no need to argue, I appreciate everyone's opinion. I mentioned this would be an emotional decision of security, essentially owning 10% more of what is currently owed, plus it pays down a little more principal automatically since my payment doesn't change.
My interest rate is ridiculously low so I understand the potential for higher gains elsewhere, and wanted to know what other conservative(typically debt averse people thought.)
As far as personal finances, we got a good deal on our home in a very nice area(Think Seattle) and the cost of living is what it is, and is far different than most areas of the country. When we moved to this particular area my wife's salary doubled and mine stayed the same. Many people send kid's to private schools, but we live in an area now that avoids that need or want since the school system is so desirable. Essentially their public school is paid for in home price to live in the area.
It's a hefty mortgage no doubt. We have zero other debt, and don't spend much on vacations and stuff. Just enjoy the home. Also, I certainly don't feel like I have to defend our choices, but some people live in other areas that cost a lot more to live. In this entire PNW region a million dollar home is much more the norm than the exception.
I like the idea of just putting it to work as mentioned and may have to consider that in a Vanguard index fund.