Oh yeah, being one of 3 kids in the back of a subcompact did suck. It wasn't the trip to the store that sucked, it was the multi-day driving vacations that had us poking each other and annoying our parents (and kicking the front seat? control your kids, ffs.) (Oh, right, it also sucked because my dad refused to run the AC in order to toughen us up.)
* Think about how often you're driving 3 kids all at once for an extended duration. The OP is driving <5k miles a year, with most of the driving time in the city.
* The OP takes a few 5-10 hour trips every year. That's completely doable in a Prius. If it's aggravating the first time, rent a minivan. They're spacious, luxurious, and might save your sanity on hour 6 as you're driving the Jersey turnpike.
It's just silly to buy a vehicle because you might use it 2 times/years for its intended purpose (in this case, transporting more than 5 people). That's like buying a pickup because you might have to move some shingles and drywall this year if your roof leaks. It's never happened in the past 10 years but ya never know.
