OK, so I'm a bit late to the Mustachian party, but we had an opportunity to pay our last debt besides the mortgage for 50 cents on the dollar, and since we actually have money in savings (a relatively new thing for us) we were able to take advantage of the offer. The next goal is a rental, and to get the mortgage on our residence paid by summer of 2022. Thanks Mustachians!
You're doing it wrong. (Sorry if it hurts your feelings.)
Good real estate returns come from leverage. Put a small amount down, and borrow the rest. Fully paying off your mortgage, especially if the rate is low, removes that leverage.
The amount to pay off your primary mortgage, could be used to buy a rental today. Now you control 2 houses with the same equity (cash). If you wait 4 more years to buy, you will likely pay more for your rental.
As always, make sure you buy a cash flow positive rental, and have some emergency fund to cover random expenses. Also, while leverage can juice the returns, keep it under control.