Depending on the type of houses available, there's a few things to consider
Tax difference between the two areas. Is living in the downtown area going to drastically increase your tax cost?
Are you able to get a house with enough bedrooms/bathrooms for what you want in either area? A house you don't like the carpet in, is a very different situation than one that doesn't have enough bedrooms or bathrooms. Those cant be fixed. You need to sort out your requirements for a house. How many bed, bath, spare rooms, pantry, back yard, garage, etc etc. What won't you live without? What do you need to be happy for the next 10+ years as your family grows. You need a house that fits what you want. Loving a huge kitchen, but buying a house with a tiny galley kitchen won't make you happy even if the price was dirt cheap or was in your favorite area. If you buy a house downtown (or in the burbs) that doesn't fit you, you still won't be happy.
Would living down town remove your need for using your car most of the time? Reduced travel costs would be a BIG bonus if one area was better that way. What about schools for the kiddos? Is one place better or worse? If you can walk to work, grocery stores, parks, school, and other important things, instead of having to drive thats a big bonus even if you have to pay for it in the house price
I would not put your entire net worth into your house, keep back 20-40% of what your sale price will be, as partly an emergency fund, and a bit to fix it up. You never know when a roof will get a leak, or a toilet will need replacing, or you need/want to do a larger project. A few gallons of paint, and some other small projects can make a place really shine. You can always add more money to the mortgage later as you work and have funds available. Make sure its a flexible enough mortgage to allow this.
If you buy a place that needs elbow grease, and are good at handy things as per your first post, it can be another long term investment in a way. Buy it low, fix it up, and increase the value for down the road. I bought an ugly duckling of a house, and now because i've worked on it, its exactly what i want and i love it. But i would never again buy a house with a galley kitchen, and without a garage. Those are two important things i will not compromise on down the road even if i had to pay more.