Our first home was 1700+ sqft, 2 stories, so it seemed smaller than a 1-story same size home, it certainly felt that way. Before we had our first baby, no problem, we had enough space, then the baby came, we no longer had enough space. We did get rid of a lot of stuff, but still we were stressed out because of the space situation. We didn't really care or hear anything other people said, so whatever they said didn't affect our decision to stay or move. Almost 20 years later after living in that small house, we built a new house, which is now our forever home, 1 story.
I think your situation may change in the future, your decision may be very clear to you if you wait. If it was me, I would not be buying a house right now, but we're waiting for the market to crash, then labor and materials will be cheaper, we'll wait til then to build an extension.