Paying off current mortgage and planning for future home

[icyappraiser]

I currently own a small, 2/2 home. My fiancee and I plan to have 2-3 children with the first one happening ~3 years from now so we'll eventually want a larger home (4BR), sometime after having our first kid. I have a number of competing desires when it comes to this:

1. Would like to continue building liquid NW towards reaching $1-1.5M FI in ~5-10 years (currently have ~300k liquid NW invested between us)
2. Would like to pay off my current mortgage (not a big fan of debt, even though I recognize this is sub-optimal)
3. Would like to have a large down payment for our future home ($100k minimum, preferably more) and flexibility to make an offer without contingencies (such as the sale of our current home)
4. Would like to increase productivity of current large cash balance

Details
Current Home Value: $210k
Current Mortgage Balance: ~$70k (3.5% 15y fixed, current payoff date 2031)
Current Cash Balance: ~$70k (would like to keep at least $10k as EF)
Current Taxable Brokerage Balance: $35k
Current Combined Income: $190k

Future Home Estimated Purchase Price: $400-450k

All options assume we would max tax advantaged accounts before paying off any mortgage balance.

[HPstache]

I think you should pay minimum and save $70K-100K.  What you do with this amount will be for you to decide down the road.  Still feeling good about paying off the house?  Do it in one fell swoop.  Would rather use it for a DP?  Well, you're almost there.  Suddenly realizing you love your home and really don't NEED 4 bedrooms?  Throw it in the market. 

Keep some flexibility, you have no idea what the future brings.  What if you and your spouse find out you can have children?  What if one has a disability that requires a special home?  What if you have a terrible birth experience as a couple and the child does nothing but cry and sleep for 2 hours at a time for 12mo and you decide you want to call it quits after one? There is so much that can change, it's good to plan like you are, but there are so many unknowns.  Save up and make the decision down the road.

[SwordGuy]

You won't need separate bedrooms for some years after the kids are born, so you have plenty of time.  Use that time to learn how to buy real estate cheaply like real estate investors do instead of buying one at retail prices.

At this point, don't pay good money for a house you don't need (and may never need for a host of possible reasons).

Paying off a mortgage at a low rate early when you plan to sell soon makes no sense.    It will be paid off when you sell.