Two deeper issues may explain why people disagree about this I bet...
First, some of us try to employ modern portfolio theory in our investing. And if one does that, you're going to include multiple asset classes including bonds (probably ideally riskless bonds)... For folks like me who think this way, early mortgage repayment can in effect become part of the bonds component of one's portfolio. (Especially true when talking about a mortgage being amortized over 10 years.)
Second, regional differences probably exist. E.g., in some areas defaulting on the mortgage may ultimately mean a deficiency judgement so someone can't simply "walk away" and let the bank eat the equity. Also, I think in some locales including mine, refinancing an $80K mortgage would be expensive in terms of fees, etc. (In my area, e.g., I think the mortgage company would push borrower to borrow more given fixed loan fees.)
A postscript: I didn't first time I read, but when I re-read the original post, I now think co-movement's portfolio probably principally includes his business (which is a great investment hopefully and should, fingers crossed, beat what he or she can do in an index fund or real estate)... and then also the real estate (which recent studies have suggested can do just as well as equities but with half the risk as discussed in link below). But if someone isn't pretty diversified and well-capitalized, I would think first balance sheet thing to do is build up some cash for safety and as opportunity fund... and then stash savings into a pension... and then, sure, pay down that mortgage.
Oh, here's the link to the referenced federal reserve research paper:
https://www.frbsf.org/economic-research/files/wp2017-25.pdf