First, congrats on being in a good position to FIRE by 50!
Second: when you are spending lots of money on more than you need, math says the sooner you cut that back, the better. The issue with real estate, though, is the transaction costs. In particular, would you downsize now to your permanent retirement house? Or would you downsize to a medium-sized place now, and then potentially downsize even more and/or move to a different area once the kids are out of the house? If you are looking at the first option, by all means, downsize now and buy a new house with 20% down and invest the rest.* If you are looking at the second option, you need to do the math on how much you would save vs. the extra costs of buying/selling/moving twice.
*Yes, what's going to happen next with the market is unknown. Guess what? It's always unknown. Buy now and you are buying the same companies you'd have bought 6 months ago, except now they're on sale. This is called "timing the market" -- almost impossible to do on purpose, but if you can luck into it and buy in a slump, you should be jumping up and down with joy.