We used to live in a 1100sf house as a family of 6. It was pretty cramped, but we *did* make good use of the space. Here are a few thoughts:
1) Saving $100k @ 7% means $7,000/year that you could put towards hotels or Airbnb when you have guests. So from a purely financial perspective, it's pretty clear.
2) I'm a strong proponent of kids sharing bedrooms. They'll probably be sharing a room most of their entire lives--roommates in college, spouse if/when married. Bunk beds are your friend, and you can get extra long twin mattresses. Of course, I'm biased, since that's how I grew up. I had a private bedroom for one year when I was in college, and while it was nice, it was definitely a luxury.
3) I agree with others who have said "combine the guest bedroom and office." Alternatively, I like the idea of repurposing a walk-in closet as an office. In our 1100sqft house, we used the formal dining room as an office, but that wouldn't work for you, since you need something separate. At one point, we put a desk in the corner of our 14x14 master bedroom, and that was plenty of room.
So if I were you, I'd stick with a 3-bedroom and laugh all the way to the bank. But you'll have to make that decision yourself.