I moved to a "nicer" neighborhood almost five years ago. We had a 3br/2ba house in the old neighborhood, and had three kids with an awkward mix of ages/genders. We wanted something with 4 bedrooms since we didn't want to pair up any combination of the kids. I have mixed feelings in hindsight, but there had been increased crime in the neighborhood, including our car and garage being broken into and finding a knife out front afterward. We looked for a 4 bedroom in our old neighborhood and there was nothing for sale for months; the market was just starting to pick up around that time and nothing stayed listed for long.
So, we moved to a nice new construction, McMansion-ish (houses don't fit most of the criteria on "McMansion Hell" aside from the size, hence the -ish) neighborhood. Cut our commute in half (4 miles with monster hill on super busy road, to 2 flat miles). I LOVE my neighborhood, but I wish we had gotten a house about 1000-1500 square feet smaller than we did, which is what we wanted, but there are a number of reasons why that didn't happen: awkward floor plans, too busy of a location, one we loved but asking way too much, etc.
Almost five years down the road, I still have mixed feelings. I really love my neighborhood; we hardly ever leave our little community. Much of our community is comprised of engineers because of the tech nearby, so we see a mix of new Teslas and old Hondas (with a smattering of BMW, Lexus, Mercedes in the mix). Our community is VERY kid-centric, so there are always kids at the park, active sports, etc. There is a much higher end community to the west that is not as kid-friendly, so we have met families in our neighborhood who are definitely able to be in that next higher tier but chose our neighborhood for the kid-friendly nature of it. Our neighbors are mostly pretty down to earth. I actually think because many of our neighbors are pretty secure financially, you don't see so much the showy displays of "wealth" that you sometimes see when people are trying harder to appear wealthy. The schools are extraordinary, and a lot of it is because of the community support, both financial and volunteer time.
I do NOT, however, love our giant mortgage and property taxes. When the oldest goes to college in a couple years I want to downsize to a townhome closer to the high school. We do our own cleaning and yard maintenance, and it takes a lot of time. I would highly advise reconsidering the smaller homes in the nice neighborhood and see if you could find a way to make that work. I thought we "needed" more space, but we definitely overdid it there and I look forward to pulling back. I recommend taking a hard look at all your "stuff" and asking yourself whether you really NEED it or not, and consider downsizing the stuff before you consider upsizing the house.
Good luck with your decision. I know your quandary is not common around here, but at least you are asking here and considering the full impact before making a decision.