The whole "living under a bridge" thing is enough to err on the side of being to cautious.
So pay your house off first. :)
If things go badly, you can sell your house, and buy a van/RV/trailer/etc, and move wherever you want.
==========
I'm never really planning to "retire" fully. But, in my lower 30s, I am making concrete plans to configure our life (me, wife, daughter, possibly more kids) for freedom. I intend to work less, and do it remotely. 20h/wk would be perfect. That leaves a lot of other time for raising kids, homeschooling, other hobbies, gardens, fish, etc.
One of the big things I'm aiming for is a productive micro-farm (or fancy gardens, whatever you want to call it). I'd like to be able to grow a good bit of our own food, and preferably have a surplus. I figure this will take 3-4 years to bootstrap in terms of raised beds, soil, compost, etc, but will then be reasonably low effort and high yield. I want to play with aquaponics as well - growing fish & plants, together, in a fairly closed loop. High yield of home grown fish in a greenhouse, year round.
I get to mostly eliminate commute costs, largely eliminate food costs over time, and have a tradeable surplus (if all goes well) that I can use to barter with other people. I get to raise my daughter outside, and we can do a lot of camping/traveling - I'd rather tour the country to homeschool than have her read books, because things are so much more interesting being there!
This isn't a thing I'd really thought was possible until spending a bunch of time on this forum and realizing that, no, I really don't need a new motorcycle every few years. Even though most of my peers are buying luxury cars and Teslas, I don't need one, because it doesn't help my quality of life.
And, really, I'm not planning to have a million dollars. Not even half that. I want some buffer in the bank, and then I can reduce my work hours to match my needs. I'll probably still have plenty of money coming in for our needs, since we just don't like doing things that cost a lot of money.
Were I planning to have a hard and fast "I retire on XX/YY/ZZZZ" plan, I'd probably be a lot more cautious. But since I'm planning to just taper back and have one foot still in the professional world, I'm just not that worried about it.