OP, pm me if you want more details.....
My situation is not terribly different than yours, but let's back up to 2003. That's when we bought our town house for about $383k, a block away from the Beltway and a Metro station. We lived there for about 5 or 6 years, then the military moved us elsewhere. We kept the house, and have refi-ed a few times, to our current rate of 2.625%. Given that we are still paying on the 2003 price, but rents and property values have gone up and I charge the current market rent, we come out very cash-flow positive (as long as I have quality tenants who pay on time). We recently moved back to the DC area, and bought another house as a primary residence. This one is expensive by my standards, but about half the price of anything else in my neighborhood. So I have a lot of mortgage debt, but it's not a problem.
If the market keeps creeping up more than general inflation where we live (and I suspect it will), once we sell the primary residence (10-15 yrs maybe?), we should have a boatload of cash. In the meantime, I plan to hang on to the rental townhouse for a long time. Hell, we may move back there once the kids are outta high school, which eventually leads to big savings on capital gains if we sell it later.
It has worked for me, and I don't see a problem with what you are envisioning. If you can get in now while the interest rates are low, when you move out you won't have to refi later, which costs more when it's not your primary residence.