So, while looking at some long term college planning for the kid, I found a time when it makes sense to pay off the mortgage.
Once I approach FI, taking a lump sum out of investments and paying the mortgage off in one fell swoop reduces my investment gains, but it reduces my annual expenses so much that it would actually push me into financial independence about 15 months earlier.
On top of that, reducing my expenses by paying off the mortgage while being retired reduces the EFC on the FAFSA calculation. That falls under the unethical ways to save money category, but it's something to keep in mind.
That said, during accumulation years, it still makes sense to pay off the mortgage as slowly as possible, and the calculation I made would still reduce long-term portfolio value, but it should get me to FI faster.