Your taxes due to the fact that the rate is lower than the AFR seem high. The difference between the AFR rate and yours is $630 in interest per month. For that to cost you $300 in taxes (a 47.6% all-in tax) seems excessive. Also, why isn't the long-term AFR of 3.68% applicable instead?
All that aside, it appears to me that we can consider this a regular loan with a 2.9% interest rate [(300*12)/265,000+1.549%].
I just checked our latest inflation numbers and it's about 6.4%. What that means is that you could have bought a few tons of raw material last year for $265,000, and sold it today for $281,960 - but you wouldn't be any richer. That material held it's value while money stuffed under your mattress lost it.
But you have a loan with an interest rate below inflation. What that means is you essentially have a faucet pouring cash into your net worth at the rate of $861.25 per month. (6.8%-2.9%)*265,000/12 = $861.25. You're asking if you should turn this faucet off, the answer is no. All you have to do support the faucet of cash is have the $625K in something that tracks inflation (like a few tons of steel) not something that loses to inflation (like dollar bills under your mattress).
But the tons of raw material as steel, or lead, or copper just sits there, and doesn't produce anything. It'd be much better if you could buy something like a $265,000 CNC machine that could be sold in a year for $281,960, but also produced some widgets in the meantime to give additional income.
If you haven't guessed, I believe what you have invested in the S&P 500 and in Apple is analogous to the CNC machine. The ownership percentage of factory, warehouse, railroad track that it represents will keep pace with inflation, and they'll continue to be used to create things, ship things, and store things. Sometimes everyone is too excited about stocks, and sometimes they're too gloomy about stocks, so it's statistical. I can't predict that next year everyone will be just as excited about things as they ices iare this year. And because prn the short term are somewhat driven by excitement or gloominess, I can't guarantee the price will be reasonable. It could be unreasonably low or unreasonably high! But the math doesn't support paying off a 2.9% mortgage when inflation is running at 6.4%