Your $600k could be invested today in SGOV and earn your $32k income gap risk-free. By adding a little risk, you could lock in about a 6% return and make that $36k. The passive income part is easy these days. Click the "buy" icon and mark this goal accomplished.
However, you also want a paid-off mortgage, and this is where the troubles start. First of all, does it make sense to prepay even a dollar on a 3.65% mortgage when risk-free investments are yielding almost 5.5%? I.e. would you take money out of an account earning 5.5% to put it into an account earning 3.65%?
So let's consider two paths:
1) You POYM, end up with no assets other than the house, and get your spending down to $50k/year. Your next task to FIRE (the presumed goal, although you didn't state it) is to save up / invest up a 25x FIRE number of $1.25M starting from zero.
2) You keep the mortgage and try to FIRE on spending of $86k/year. Now you need a 25x FIRE number of $2.15M but you have $600k in seed money to start with. Thus you need to save up / invest up $1.55M.
#1 might superficially seem like the fastest path because you can apply the $36k per year mortgage savings to building the nest egg, and because you have a lower 25x FIRE number. However, you miss out on the compound returns from your $600k in assets which largely negates these benefits. Also, #2 has hidden safety features not reflected in the napkin math: After year 28, a big chunk of your spending goes down, which means that if your portfolio is struggling at this future time, there is relief ahead. Second, if mortgage rates ever go below your 3.65% rate - as unlikely as that may sound today, it could happen - you have the option to refinance. Third, you end up wealthier when you FIRE in scenario 2 than when you FIRE in scenario 1. Finally, #1 is risky because paying off the mortgage now would leave you at risk if a big expense came up or if your career/income did another cyclical downturn. You'd be house-poor for a while and completely exposed to both the risks of your local real estate market and inflation.
I did some Excel math with the following assumptions and simple annual contributions and compounding:
Scenario 1: 5% returns, $0 starting point, $136k per year contributions, FIRE number = $1.25M
Scenario 2: 5% returns, $600k starting point, $100k per year contributions (because mortgage reduces amount you can save), FIRE number = $2.15M
Scenario #1 exceeds its FIRE number in year 8 and scenario #2 exceeds its FIRE number in year 10. However, by year 10 your scenario #2 self has about a half million dollars in liquid assets more than your scenario #1 self. This is partially offset by the presumed remaining debt on the house of perhaps $450k. I tried manipulating the rate of return - maybe you invest it all in stocks and earn 10% per year? - and at a 10% return both strategies reach FIRE at roughly the same time, in six years.
So Scenario 1 involves taking more risk (being almost broke for a while, putting all your eggs in one housing market) and giving up a great interest rate on debt that is lower than today's risk-free rate of return, but it can be expected to technically get you across the 25x finish line faster.
What I actually suggest is a hybrid approach that is the best of both options: Invest your $600k until you reach what would be your FIRE number if your mortgage was paid off plus your mortgage balance, then pay off your mortgage and retire.