Remember, every day that you are traveling is a day that you're not at home, so you're not buying food at the supermarket, not using your air conditioning, not spending from your entertainment budget, not driving around in your car, etc. So, your travel budget is partly offest by reductions in those costs.
In fact, my experience is that I spend about the same whilst on holiday as when I'm at home, without factoring in the airfares. That's based on a mix of mostly cheaper travel in Asia, Mexico, etc., with some US and Europe (maybe 30%) thrown in to the mix.
Hence, taking advantage of frequent flyer miles, airfare sales, discount airlines like Spirit and WOW, and off peak travel can make a huge difference. For example, I wanted to visit my cousin in Seattle but I waited until I found a round trip flight on Spirit for $193. Being FIRE gives one the flexibility to take advantage of that kind of opportunity.
The way that I budget for trips is to come up with a travel budget that includes all travel expenses (this year it'll be $11,000 for 12 weeks of travel in six trips - Kyrgyzstan, 2 weeks, $2500; Mexico, 1 week, $1,300; Dublin, 1 week, $1,200; Vietnam, 2.5 weeks, $2,000; Indonesia, 4.5 weeks, $3,000; Zion/Bryce Canyon, 1 week, $1,000) but to do my other expense calculations on a weekly basis and to come up with my annual numbers for those categories based on the number of weeks I'll actually be at home (40).