I read your post and think that you are asking the wrong questions. You and your wife first need to decide how you view money. Money can either buy you stuff, or money can buy you freedom. Life WILL throw curveballs at you. Companies go out of business, or merge with another and you find yourself being an extra employee, leading to being not an employee. Home appliances suddenly fail. Health problems pop up. The list is endless. Having money set aside frees you from worrying how these curveballs will be handled.
Your family NEEDS food, shelter, and loving parents that will prepare the kids for adult life. The rest are wants. As someone who grew up in abusive and dysfunctional homes, then in the CPS system for a while, and finally good parents, I am most grateful for the ones that actually spent time with me and cared, even though we lived in a 900 sq ft 2 bedroom apartment. Big houses and nice cars never entered the equation. I'm actually grateful for growing up poor because now I know how to fix things myself and make things last, and I have memories of spending time with Dad under the hood of a 69 Malibu wagon.
If you and your wife decide to choose freedom over stuff, keep the smaller house, pay off the loans, and focus on increasing your net worth. Get rid of the debt as soon as possible, interest is nothing more than paying extra for something in order to have it now. Use Dave's snowball, or highest interest rate first, it really doesn't matter as long as you are making the debt go away.
If you and your wife decide to choose stuff over freedom, it's just a matter of playing a shell game of moving your debt payments around, hoping to finance your way to prosperity. Refinance, consolidate loans, pay other debt with home equity.
The formula is easy. Want it now = pay interest = everything costs more. Want to increase your net worth = delay the purchase and save for it = everything costs less