Regardless of budget, I will say the following:
We found that layout matters more than square footage. Our first house was ~1400 ft2, and was plenty of space, but we still felt very frustrated with it because of the layout. We had a ~100 ft2 closet because the space couldn't be used for anything else, and a ~200 ft2 playroom because we couldn't do anything else with it. In addition to that, the house was two small, square floors, meaning there was no separation/privacy. Our new house is 3 floors and the separation is much better.
Our new place is "only" 1900 ft2 (not much bigger than the old), but feels HUGE because the space is much better utilized. (I asked my father to guess the size after we moved in, and he guessed 2500-3000 ft2). Multi-use spaces make a big difference here: an in-kitchen dining room, but still enough space for a 6-8 person table, plus a 2-stool bar; an extra wide hall that looks awkward at first glance, but actually works quite well as an office with a very large desk; ~400 ft2 of under-garage, unfinished storage/work space; and a feature we love- the garage door opens directly into the kitchen, which is fantastic for groceries ;-)