Absolutely prioritize location. Everything else can be changed. Small is also good - it costs less in every respect to maintain a smaller house. Plus, you can keep a better eye on the kids! As long as the structure is sound, other changes can be made over time. Don't forget that anything that's the rage now, will be dated in 10 years anyway. Think "vintage" instead of outdated. I'd rather have a genuine 1930's house than a genuine 2000 house.
This! We prioritized proximately to DH's workplace over anything else. We bought a 1700 square foot vintage 1920s house with 3 bedrooms but no separate family room. It had been "remodeled" 11 years prior, and it was so dated! The kitchen had blue formica counters. The front bathroom had an 1980s makeover including a shower/bathtub insert with brass shower doors, as well as a pony wall and an apartment quality sink cabinet, and it had been sponge painted with pink and burgundy. The intended master bedroom, done in 1991, had floral wallpaper I'm pretty sure wasn't even current when it was hung, and light blue carpeting.
Over the years we put in sweat equity and also had some refreshing done -- the most expensive in house remodel was skinning the kitchen counters, replacing the cabinet doors, and putting in new countertops, and that was about $5K. DH and BIL gutted and redid the front bathroom, and now it will never look out of place in our vintage house. But one thing we've learned is you either have to accept that things get used and will eventually look like it, or you throw money at it over and over again. One nice thing about a nice house is it would never make sense to try to remodel it to look like a newer house, so we don't feel the pressure to make it look like a new or flip house. The era of the accent wall never touched our house. We get away with having less than perfect wood floors, because they're almost 100 years old.
Over the years we've switched up room functions, and we always make it work, even if it's unconventional -- that master bedroom is a work-from-home office by day, TV/movie watching area in the evening, and bedroom for a young adult by night. On weekend days it's mostly a craft room and music room. It's not perfect, but it's ours. For 25+ years DH has been able to commute by foot or bike (previous condo was in same neighborhood), saving so much money, but also giving us all more time together. Yes, we feel a little squeezed at times with four adults in one not-so-big single story house, but within the decade it will seem much bigger when it's only DH and I. I'm certain we'll be glad not to have 2500 - 3500 square feet of house to clean and maintain.
We live in a HCOL area and chose a house not quite double our gross annual income. People had opinions about us choosing a smaller house in a less "family friendly" neighborhood (which I guess means in a planned community or on a cul de sac). But we weathered every economic downturn without fear of losing our house. As friends moved "up the hill", we stayed put near our city's downtown, within walking distance of many amenities. Buying within/below our means has never been a mistake.