I love the idea, but not with the current property and financing structure. You have a lot of sunk costs and it would be nice not to incur selling expenses. However, over time, you will have significantly negative cash flow from this property. If you had a mortgage payment that was much lower and you were cash flow positive, and argument could be made for keeping it, although you could probably get a better ROI on something else. Add the cost and risk of the pool in, and it's a no-go for me.
It can be very profitable to buy a house as an owner-occupant and after a couple of years, convert it to a rental. Move into the next future rental as your primary residence, then rinse and repeat. Your rentals are financed at lower owner-occupant interest rates, a significant advantage. To succeed at this plan, you have to buy and occupy properties that will be good rentals in the future. Generally that means bread and butter three and four bedroom houses with good schools but no expensive frills and no pools.