I think it depends on both the parents and the baby. For us, even though we like travelling, we've stayed relatively put during the early years. Frankly, it just seemed a lot easier. So, in that sense, we were being selfish. Plus, our daughter is not a great sleeper (currently 2.5 years old) and the idea of being in lots of new places / not being able to keep a schedule is pretty daunting. If we'd been traveling the whole time, though, maybe she'd be used to it. It also worked out for us in other ways -- I'm in the middle of a 5 year dharma study class, so extended travel wouldn't be supportive of that.
At the same time, I spent my twenties living in Taiwan and China, travelling throughout Asia, so I've scratched that itch. And my wife isn't particularly driven to that sort of life. If we both REALLY wanted to travel, though, it might be different. I also think it's probably very hard to know what it's going to be like until you do it.