bb john,
I'm going to start by saying i am by no stretch of the imagination, a master of this, and I actually had a post myself recently asking for advice. But here is what i've gathered just from the advice that has been thrown at me.
If your current mortgage payment is 2k a month including taxes, that equates out to 24k a year.
What you're saying your yearly expenses (roughly of course) are, equates to ~24k a year. Now, it seems like the money you're throwing at your mortgage each month could be used for your monthly expenses if it was paid off in full... as such, you would instantly be gaining 24k / year in 'savings' without changing anything.
Through the articles i've read from MMM himself, the 'average' family of four should be able to live comfortably (not lavishly...) on that amount, and there are some of those out there that live on less.
Now, granted 4% is not a bad interest rate, most of the advice that was given to me was based on a loan with 10% interest... regardless, the money that you're spending there would cover your yearly expenses, and eveyrthing else you make, part time, full time, interest on account, etc... would be going toward a net profit, slowly replenishing your 'stache technically without you doing anything.
Again, take my input with a grain of salt, as I'm by no means an expert or perfect. I would highly suggest doing some math and see if that would be a feasible choice, and perhaps take some advice from one more experienced than I, however, that is just how I see the numbers work.
Best of luck to you!