I would say no, but we did something similar. We had paid off our mortgage in 2008 and in 2014 found an undervalued rental/second home in an area I knew very well, had been looking at for some time and, in fact, where my parents owned a second home for most of my childhood.
Getting a mortgage for a second home will not happen when you're self-employed. So we re-mortgaged the primary residence. The second home only rents on a weekly basis (at a premium) from mid-June through September. We use it several weeks a year and look at it as a future FIRE residence, or at least somewhere we'll spend more time.
If we viewed it as only an investment property, as some of the owners of neighboring homes do, I would say no-way-in-hell. Not worth the aggravation.