We only have two children, but we just downsized from 2700sf to 1500sf so we could pay off the mortgage quickly and because we were wasting a ton of space. We are in the process of finishing off a playroom, but that is more to up the property value than a need for space. We moved in August and have no regrets so far. We truly don't miss the square footage that we lost although I will say that two bathrooms were on my "must-have" list.
We actually considered doing this for two years and spend a lot of time looking for the right house (our must haves were same school zone, big yard, two bathrooms, three bedrooms, separate laundry room, big pantry) at the right price (under 200K). The price point made it really hard to find and any house in that range tends to get snapped up in a matter of hours. I ended spotting the for sale within hours of it going on the market. Fortunately, it was over a holiday weekend so not much traffic, and we were able to get it. We got a great price and will spend about 25K on renovations (updated bathrooms and finishing out the bonus room).
All told this move knocked off 100K in mortgage debt and we are saving big on utilities, property taxes, and on not having an HOA.
Take your time to figure out what you need and take your time to find out. We had a contract on another house several months prior to buying this one that fell through, and we are very thankful for that. In retrospect, we were more attracted to the price than the actual house. We ended up paying less for this house, but have a much bigger yard (over an acre), it's completely brick, and there was expansion space for the playroom.