We took a total of 30k out of ours for our down payment 10 years ago. I wish we had just sucked it up and saved hard for another ~1.5 years instead. I would be a good year closer to retirement now if I had. Retirement was mythical to me at the time, and I was buying the whole mortgage debt is "good debt" tripe at the time.
My strong advise is to never buy a house you don't reasonably expect to be in or rent for at least 7-10 years. Being a landlord from across the country can be quite a pain.
Even if are buying it to be a rental, buy a place based on its rental prospects. Your dream home and a good rental house are likely very different in location, cost, maintenance, etc.