I live in a very large city now but have previously lived in (and have lots of friends and family in) small to mid-size cities. I happen to have job that only tends to be found (at a reasonable salary) in big cities so I will not be able to change until retirement. That said, I really like the smaller cities and have never been bored. What I have found--
1. Medical care is limited when it comes to specialists. There will be plenty of primary care, OB-gyn, dentists, etc but, say, ENT surgeons or psychiatrists will be limited to just a few...so if you don't like them, you're driving into the big city for care. Mental health services can be especially tough to get.
2. Most mid-size cities have an airport, but flights are often pretty expensive compared to flying into a hub. Again, a lot of people I know end up driving into the city for air travel. And public transportation tends to be very limited.
3. There will be plenty of restaurants that serve the regional cuisine or "American", and probably one or two other types of food. (My parents' city has amazing Tex-Mex.) Other kinds of cuisine may be limited to just a couple of restaurants, of varying quality.
4. It depends what you like for fun. There won't be any issue with new movies, or access to cable or high-speed internet (as there might be in a more rural area). Outdoor fun is more accessible because you don't have to drive a long way to get there--my sibs and I all live fairly close to the center of our big metropolitan areas and it's a good hour's drive to get to parks, the beach, etc. My parents can be at the boat dock in ten minutes. For the arts, the biggest shows and artists won't tour there, and the museums won't get the biggest exhibitions. But the local venues are often very energetic and enthusiastic about what they do provide, and if you want to get involved (as a performer, volunteer, etc) you'll be welcomed with open arms in a way that you wouldn't be in the big city. The tickets are much cheaper, so you could see four plays for the cost of one ticket in the big city. I know that this isn't always the case in areas that are in economic decline, but all the cities I tend to be visiting have really nice parks, libraries, and other public facilities.
5. For shopping, big grocery chains and big box stores will all be available but certain brands may not be (a lot of places don't have Trader Joe's, for instance). Amazon may take longer and you probably can't get all the delivery services you do in big cities. Uber, etc may or may not be available.
6. Of course it varies by city and neighborhood, but most of the mid-size cities I visit have solid school systems and plenty of kid activities. However, they probably won't have the variety of a big city or "elite" opportunities. (Ie, there will be ballet classes, but not the opportunity to study at the Dallas Ballet.)
Personally, I tend to get stressed by crowds and traffic, so I would have a lot more fun going to an event in a smaller city, even if it isn't the same quality as the big-city version, but YMMV.
If you tell us more about what you and your family like to do, we could give more specific advice.