On my first date with my now husband during grad school, I talked about how I liked to travel, and he looked at me like I was crazy. He had been out of the country just once before. We were both somewhat frugal at the time, but I was generally willing to spend a decent chunk of money on travel.
4 years later we married, and that fall took a trip around the world for a year visiting 26 countries, spending less than we would have staying in Chicago for the year. We were really glad we did it while we were young and childless. We did a ton of hiking that might have been harder when we were older.
3 years after that trip (we're now in our mid-30s), we now have an infant son and are getting ready to FIRE soon. We're thinking mostly domestic travel for 2016 since we have the Southwest Companion Pass and maybe some slow travel abroad in 2017. He now has the travel bug more than I do - I kind of want to hunker down with the baby while he still wants to go everywhere. We'll find some middle ground.
Honestly, rather than a specific set of ideas, I think what I liked when I met my husband was that he was adventurous and open-minded, but also rational, practical and willing to delay gratification. So he's remained frugal but changed a lot in other ways - meat-eating to now vegan, now loves travel, really expanded his palette of things he likes to eat, etc.
I think it's too rigid to say you only want to travel after you retire in your 40s, and you will eliminate women who might otherwise be a good match. Overseas budget travel can be ludicrously cheap and not incompatible with frugality/FIRE.