It's really your decision on how to handle it. I would consider these things:
If you are going to knock it out in a couple years, what is the difference in total interest paid with your current rate vs. the refinance rate? Over just a couple years it may be negligible (especially considering refinance fees).
Do you want the guaranteed return of 3.99% on your money by paying down that loan now, or the (Market return)-(3.99%) = Return of leaving your money in stocks and while saving then paying off once the balance is full?
Actually this is an even more complicated question because it is 3.99%+(mortgate interest paid tax savings) = Mortgage Payoff Return or (Market return - capital gains tax) - (Mortgage Payoff Return) = Save In Market Then Lump Sum Payoff Return. Whereas SIMTLSP Return could be higher, it could also be negative if the market goes down, or just plain negligible again. It's your choice on if you want the risk in return for the potential reward of paying it off quicker.
I am not you, nor do I presume that I know the best way to tackle this problem. However, I have also decided on the mortgage payoff route (I want the freedom from my job too), and my plan is: build Emergency Fund (5k in my case, done), max 401k (done), max Roth IRA (not quite there), then extra payments to mortgage. Taxable account investing will replace the extra mortgage payments once it is paid off. Then of course, the final step is RE.