When you look at the house costs as well, don't forget to factor in the extra costs and time (property tax, reno/maintenance, etc. Even with everything DIY, maintenance can chew up more than expected. More than doubly so if you contract it out. If the housing market were to continue to rise, you might lose out on some appreciation costs. But, if your condo costs are consistently less than the house operating costs, invest the difference and you may come out ahead anyway.
Our family of 5 (3.5 yr-old twins, 15 month-old) recently moved from a house about 1500 sqft to an apartment that's about 700 sqft. Still working on downsizing a few things, but I actually prefer the apartment. The benefit of a smaller space is it focuses you to live without a lot of "bloat". I like the ideal of minimalism in the sense of everything has a purpose, and is used frequently. It also prevents buying (or receiving as gifts) unnecessary crap.
In the house, most of my non-work time went to reno or maintenance projects (replace the front steps, reinforce a deck, etc, etc). Throw in routine maintenance (yards, snow), and there wasn't a ton of time to actually do stuff with the wife and kids.