It totally depends on the COL.
In our LCOL area, the cost of an extra space of some kind (office, basement, 4th bedroom) is fairly minimal. If I lived in NYC, guests can get a hotel.
For us, we do have guests quite a bit. My mom just stayed for a week. My dad, mom, sister, and BF are here for a week at xmas. My parents will come at least 2 other times a year. My inlaws come maybe 5-6 single night trips a year. My grandparents come once a year. Etc.
We have a 4 bedroom house. The 4th bedroom is actually a play room/treadmill room because we use those things on a daily basis. When we have guests, the kids get kicked out of their rooms, the guests get the beds, and the kids sleep on the floor. It is costing us an extra zero dollars a month in rent (3 bed houses are the same price) and a tiny bit in utilities.
If I was buying a house, would I pay extra for a guest room? Not, probably not. But I would pay some small amount extra for some kind of space outside of the main living room, to use as an office or play room or exercise room or whatever. And I might stick a bed in the corner.