Whether to pay down your mortgage or not depends upon more than a mathematical hypothetical. Although the trend has been for the stock market to go up over the long term, that doesn't mean that it always will and that doesn't mean that there won't be down periods on the march upwards. There is risk in investing in anything, whether it be your house, stocks, bonds, etc.
What if I become mentally disabled? What if a caregiver is able to exploit me and gain access to my accounts and drain them? It is very hard to recover those assets unless you can somehow pin it on the financial institution as negligence. Although it can be done, it is a lot harder for them to successfully get my house. Even if they make me sign a deed, it can be invalidated if I didn't have the capacity to know what I was signing. Then at least one of my assets would be available for my future needs.
If someone's goal is to accumulate the most wealth possible in the shortest time possible and they are willing to take any steps to do that, then perhaps a facepunch for being unwilling to consider investing instead of paying down a mortgage is warranted. However, for most people, there are other considerations and values than just the accumulation of wealth quickly.
OP, I don't disagree with the theoretical logic of your argument. However, for people who are out of debt, saving and investing, I hardly think that because they choose to diversify their assets differently from you that they should be "facepunched."