How much is in your EF and how much do you have saved for retirement?
It could make a lot more sense to beef up the retirement accounts instead of paying off the mortgage. If you want to RE, beefing up the non-retirement accounts is also a smart move. I doubt that you are paying enough interest to exceed the standard deduction, especially without a rental, but you should be able to invest and earn more than 4.5% long term even without the mortgage interest deduction.
For starters, each of you can do a Roth for 2015 and 2016 if you haven't yet done so.