My understanding of the "Should I pay off X or invest instead?" question usually comes down to rate of return on debt vs. rate of return on investments.

Yes, that's the "Math" side OP mentioned.

It's a bit more nuanced with a house. For example, the base of paying down your debt early is what you get back in interest you didn't pay (3.5%)

Yes. Nuances include tax deductibility of the mortgage interest, etc.

plus whatever your increase in property value is. So, putting $1000 extra toward the mortgage when prop values are only going to go up 1% means a 4.5% return... But if property values are going to go up 4%+, you get 7.5%, which is 0.5% better than market average, so...

Simply: (loan rate %) + (% increase in prop value) = Total ROI on extra mortgage payments

No. The loan rate % part is ok but, due to leverage, not the prop value part. For example,

- House purchase price $100K. $20K down payment.

- Principal starts being paid off. Meanwhile the house value rises to $150K, and the house is sold for that amount.

- If $5K more principal has been paid, ROI = $50K/$25K = 200%

- If all remaining $80K has been paid, ROI = $50K/$100K = 50%

Of course, leverage works against you if the house price falls.