I'm definitely in the don't pay it off camp. The math supports this and debt doesn't bother me when I have a large portfolio balance giving me the ability to pay it off any time I want. But if getting rid of the mortgage before FIRE helps you to feel more comfortable FIREing, then go for it.
As many have mentioned, you don't need to use the 4% rule for your mortgage payment. I go with the same approach as
@Path2FIRE mentioned (tack mortgage balance at retirement date onto stash needed to support all other non-mortgage expenses at 4% or whatever SWR chosen). In the large majority of cases, you'll come out ahead by making minimum payments. I did a cFIREsim calculation assuming a $250k mortgage taken out today for the next 30 years at 3.5%. Monthly payments would be $1,123, so yearly would be $13,476. If I put in $250k for portfolio value, $13,476 for yearly spending and make it
non-inflation adjusted (this is key why 4% rule is not needed) for a 30 year period and leave all other values as default (75% stocks, 25% bonds), I get a 98.33% success rate. The median ending portfolio is $366k. So you're more than likely to have more than you started if you invest rather than pay it off. But there is a small chance in the worst times, you'll run out of money. Personally, I'll take those odds, but if that tiny chance is bothering you, then go ahead and pay it off. Just realize you're likely giving up hundreds of thousands of dollars by doing so.
https://www.cfiresim.com/6bc44151-f401-4d0b-81be-2306631cbfccAs far as extra taxes in retirement, that's definitely something worth considering and modeling if possible. But like
@wageslave23 said, you can realize all your gains today whether you pay off the mortgage or continue to invest so this shouldn't really be a major factor. Any additional taxes you pay on gains shouldn't bother you. Because you only have those gains by continuing to invest. I'm happy to pay $200 tax on my $1000 gain rather than $0 tax on $0 gain. I'm still $800 ahead even after paying taxes. The ACA does complicate things but it's not impossible to model and come up with a plan that works for you.
Finally, it might make sense to refinance. 3.5% is low, yes. But I refinanced last fall from 3.5% to 3.125% for no cost on a much lower mortgage balance ($115k or so). I used LenderFi and they were quick and pretty painless. Not sure if the deals are still out there, but if you can get a true no-cost refinance (get a lender credit to cover title and other fees), then it's a no-brainer. Obviously a lower rate will make the mortgage even more advantageous.