I am 48, and my son is 4. My wife and I definitely went through thoughts / discussions / planning for these considerations. I joke that DS will graduate college when I turn 65, so we have timed everything perfectly! (by traditional standards--old tradition, even) We *might* be RE now--I am working through that. I just got a redundancy package, so I'm not working, and got a fat Europe-level payout to deploy. But for our schedule, it's as if we are FIRE now; the only things scheduled are appointments we make, or DS's schedule.
The first thing to think through, retired or not, is that as an older parent you have more resources, but likely less energy than typical. That meant we didn't blink at having a (part-time) nanny for DS's first 2 years--we could use the help! And we could handle the cost. It also meant that we stuffed his 529 plan to the max tax deduction, and were done with it as of last year. (funded $50k, starting at year 0, with an expectation of growth to $100-$150k)
So, financially it added complications, but I can't really say it delayed our plans specifically. Who knows what we would have planned by now, otherwise? We did what we wanted to, which is the real core thing about FI.
You do bring up a great point about time, though. We are about to be anchored to some kind of school calendar for 12 years. Bummer. Can't even sleep in, in the mornings. At least for a few years. He is already in kindergarten, so I get up at 6:15, along with him, to get ready and commute to school. (school bus service is rare, and through private providers if offered, in Dublin; we take the train)
When we return to the US, we'll be headed to Houston for family reasons. (I've never lived there before; DW moved away after high school) And we are likely to move there in the summer. As a lifetime "northerner," I've tried to wrap my head around the fact that in Houston, summer is winter: that is, summer is the season that impacts what you can do, and whether you want to spend time outdoors.
So, it leads me to my first suggestion: you mention heading to warm climates in winter. How about *living* in a warm climate, and visiting your current location for the summer, when the weather is good and all of you are free from schedules? For us, I imagine a fairly regular migration through MI and MN to visit family, slow traveling to stay connected to places we know, but with the freedom to go somewhere new each year, too. And it can be 3 months long!
The other option you would have is some kind of home schooling. This doesn't need to be a permanent commitment: you could do it for a year or two. My FIL's parents pulled him out of school for a term to tour the national parks in the West. In the 50's. The school was quite cooperative for the opportunity. Who knows if your school would cooperate as much as they did, but they also probably couldn't stop you.
Our idea of travelling regularly addresses the only real regret I have for being an older parent: it limits the time DS has with his grandparents. Who knows if they will see him graduate HS or College, or get married. But I can make sure he remembers them, and we build memories as long as they are along. They are spread out quite a bit, so the travel is necessary to connect with all of them. (more than two, due to divorce / remarriage)