Before I made a decision, I'd do some more calculations. First, make sure your rental income estimate is accurate. Second, look at what a low- or no-cost owner occupied 30 year mortgage at 75-80% LTV would cost on a monthly basis. (You can use the cash you're pulling out for a DP on your next property*). Then run the numbers again and see if you like the results better. Rushing to pay off a 3.5% mortgage on a rental property could be a bone head move in your situation . Plus, the cheapest loan rates are for owner occupied properties. Once it closes, you can move out, but not before. No, the bank will not care, just as long as you keep sending them timely checks.
Possible better options: Refi as above, rent the house for about 2 years, then sell it. The IRS will let a married couple keep up to 500k profit from the sale of their primary home TAX FREE, provided they lived in the house for two of the last five years. Then move that money into other investments. Yeah baby, that's making the house work for you! Tax free monet is amazing!
I understand pride of ownership and fear of renters who trash. You're going to avoid those problems by cherry picking your tenant. Begin with refusing to rent to anyone with pets and thoroughly checking all references, including credit. Then do your due diligence as a landlord, including collecting the largest deposit the law allows, doing regular inspections and maintenance. You should be fine.
*Technically, no you cannot borrow money for your DP. Typically, the bank will want to see where the money came from. What they usually want for documentation is two months of bank statements. If you time your deposits well, you should only have to have the money in your account for a little over thirty days to fulfill the seasoning requirement. BTW, if the bank asks why you want to do a cash-out refi, you can say you want to do more home improvements or pay cash for a car. Because you're not a consumer sukka, that's not actually what you'll do with the money, but as long as the payments keep rolling in, the bank won't care.