You are assuming that you will get what the average return has been historically in the past. You are assuming the returns in the past will look the same in the future.
The past may or may not look like the future, many believe it will. It probably won't be in the short term. Your gamble could work, but only if you don't need that money for sure for the next 20 years. If you need it in the next 5 or 10 years, it could be worth less than it's value is now. The markets are funny, and we've been riding a bull for a while. You may see some volatility in the coming months or years. That may be hard to stomach with borrowed money. Personally, I wouldn't do it.