Other ideas that might be a factor perhaps (in no particular order):
-a house in the higher end of the market is usually bigger and thus requires more maintenance. This costs time and money, affecting your savings rate pre FIRE.
-big house pre FIRE is more expensive on utilities, all other things equal. Affects savings rate.
-minimalists are often a fan of small spaces as they make it easier mentally not to fill them with stuff. Not buying stuff saves money. Remove the urges.
-If you get into a smaller space now, you have more time to prepare your living situation there for FIRE.
-I reckon if you made the call to sell a million $ house now and downsize for FIRE, it would be, obviously, a big call to make if you asked any normal spendypants. But, making such a big change, it is a BIG statement to yourselves, that you are clear about your goals to FIRE, and that you will achieve it. You are prepared to do such things in order to FIRE. Doing it now removes that big decision from the motivation/procrastination equation and simplifies things moving forward.
-Being in your smaller FIRE house, you feel and know what its like to live in it, in that exact space. You can much more easily mentally picture Post FIRE.
-Being in your smaller FIRE house, you have time to prepare and invest in things that are moustachian but dependent on that location. Examples could be, making it more off grid in various ways, planning building and using a veggie garden, buying in bulk spare compressors on the cheap for that HVAC unit that is specific to the house (random, yes, but...). If you are in the bigger house, its possible you will put these things off. An enormous veggie garden with fruit trees, for example, might not be desirable in that suburb when it comes to sell. Also the fruit trees take time to establish. Especially in the few years prior to FIRE, some of these you might put off, affecting your savings rate pre FIRE.
-Diversification. If you keep it until FIRE, you would have over half a million in a single asset in a single location. This is a normal issue with real estate investing and is up to you whether that risk is an issue.
-Society might change a bit at the higher class end, who knows. Your million $ house may be less desirable by then. Its more dependent on a small market (in terms of numbers of buyers). It might be more desirable. Many property investors tend to avoid the higher end of town for various reasons, but often it can be more variable in both directions.
Just some things to think about :-)