A bigger car also allows your kids to take friends along for in town trips or when car pooling starts up. Not necessary, but it is another practical use for a larger car. We managed with a small car (versa) when they were infant/toddlers, but upgraded to the minivan when it died. Not necessary, but nice if you need to add another kid or adult in the car.
I agree with this -- infant/toddler crap is bulky and annoying, but manageable. But when they get older and want to take friends places, it's nice to have that capability.
But for me, that's another reason to stick with the small car for now -- keep the costs low until you actually have a regular use for that additional space (plus have more time to save up for the larger option), rather than buying more space which your crap will promptly expand to fill.
FWIW, I read the comment about vacations in the old days as focusing on the comfort level of being squished together and surrounded by stuff, not as disparaging modern safety requirements. I don't hear anyone here saying to skip the carseats, just that you don't actually
need an adult in the back seat, because we all somehow got by without one. And frankly, most of the pain and annoyance from traveling with small kids is not something that a bigger car can fix; the kid's going to cry, my DS is going to need to go potty 15 minutes after we leave, everyone will be bored and cranky, etc. etc. etc.
OP: worth looking into flying just for the annoyance factor, if you can work out the car thing on the other end. I can drive 6-7 hrs with the kids to my dad's (including navigating traffic around DC, a/k/a "leave at an ungodly hour and cross all of your fingers and toes that it doesn't take 3 hrs"), or take a one-hour flight for @$120 apiece round trip. For me, that's pretty close to a no-brainer.