A few things change when you FIRE, and all of them make a mortgage less advantageous. First, you will have less (if any) taxable income, so any tax benefit of mortgage interest becomes less valuable. Second, your risk tolerance should get lower, because if something goes very wrong its harder to jump out of retirement into a new job than it would have been to keep your job in the first place -- you are more vulnerable.
Third, and most importantly I think, you no longer have a regular cash inflow from your salary. Instead, your cash outflow needs must be funded by investment income. Some of that investment income may be in the form of dividends and interest payments, but likely some of it is in the form of selling assets. Instead of dollar-cost-averaging into the market (which means you're buying more shares when they're cheaper which is good), you are dollar-cost-averaging OUT OF the market (selling more shares when they're cheaper which is bad).
Your increased vulnerability (higher risk aversion) and the difference between dollar-cost-averaging IN versus OUT are two big reasons why the SWR is 4% and not 6%.
So your investment portfolio can fund a stream of cashflows equal to 4% of principal. What can your mortgage fund? A stream of cashflows equal to the future interest payments you make. If your mortgage interest rate is above your SWR, you should prepay the mortgage when you FIRE.
If you choose not to do so, you will be better off ON AVERAGE! This is counter-intuitive, why did I just recommend doing something that makes you worse off on average? Because your retirement planning efforts don't focus on the average case, they focus on protecting you from the worst case. In a worst-case scenario, the market crashes early in your retirement, and your investment portfolio is down 50%. You have to sell twice as many shares to fund that mortgage payment because you are dollar-cost-averaging OUT, and when the market recovers back to its 7%-annual-appreciation trend, you don't fully participate in the recovery. Your future income is impaired for life because of market volatility that would have made you wealthier, not poorer, if it happened before FIRE. This risk is lower if you prepay your mortgage.
Another way to demonstrate this is as follows -- if your mortgage rate is 5% on a $100,000 balance, and your SWR is 4%, you will need to save $125,000 in your investment account to give you enough reliable income to pay your mortgage interest. If you instead use $100,000 of that money to prepay your mortgage, you'll have $25,000 and zero debt left, and that $25,000 will generate $1000 of reliable income throughout your retirement that you can use on other things.