Willn I understand your point to a degree, except I interpret it differently.
If you are blowing money, then yes pay down debt, don't blow the money. If you are FI then why take any risks even if mathematically optimal. If you are fresh out of school, have great jobs, have costs under control, then rock the Stache!
If you are systematically investing the money, then I think you will be mathematically safer with debt at 3%.
You named off a few categories of risks.
Job loss, health risk, black swan event, freedom to change jobs, etc. I would rather have $350,000 in investments and a 30 year mortgage at 3.5% and a $1,123 monthly payment then a paid off $350,000 house and no investments. The flexibilities of investments is crazy. Having all your eggs in one basket (house) is riskier.
The best thing that could happen to me is Carter like inflation. Inflation is the greatest thing ever when you have a 30 year fixed rate loans. The stock market does great, your other investments rock, and your mortgage is fixed! Rents go up on your rentals to reflect the infation, but the government and banks locked you in for 30 years. Glorious times for a financially independent person who has 30 year fixed rate loans as a hedge.
If you want to see how the market did during various presidents check out this old article.
http://www.forbes.com/2004/07/21/cx_da_0721presidents.htmlI am good with people paying off debt if they understand why, whether that is emotionally or financially. I just push back, when I hear pay off debt as a mantra that debt is bad and needs to be killed. I don't think that is always the case. The rules of thumbs/wives tales, etc. that have been fairly accurate when mortgages were 8-20% worked very well. When rates are below inflation and below projected inflation I don't think you can just go with it is safer with no debt. If you believe this then make sure your Safe Withdrawal Rate is below 1% or based on life expectancy as you are saying that over the next 10-30 years we are going to have investment returns less than inflation and less than anything seen over the past few hundred years. Firecalc and all the models out there will not hold up.