Everyone likes to point to pro athlete spending, which can be ridiculous, but I remember reading an SI story interviewing guys and they point out that it's not so much the spending as the hangers-on. Let's face it, if you buy a $200k Lamborghini in cash, after a year or two you still have a $100k Lamborghini you can sell. Same with your mansion, it's still an asset, and as long as you didn't finance it (a lot of these guys are at least smart enough to pay cash or put a lot of cash down) too badly, you can sell it and get out. Lose money, but you still have SOMETHING.
No, what really hurts is that everyone in the inner city that you grew up with, parents, siblings, aunts/uncles, buddies, etc, all see you as a winning lotto ticket and demand you support them or invest in their "business ideas" and all that. That's where a lot of these guys throw their cash and they never see it again.