I liken any enthusiast hobby to drinking: "if you never try it, you'll never miss it".
I could never spend that type of money on any electronic device even though I am somewhat of an audiophile myself and can definitely hear differences between 128kbit and 320kbit, differences in speakers, etc...
However, I think this mentality sort of grounds an "I can justify these purchases because I can hear the differences and not everyone can". That's a bad mentality to fall into, though I think (sub)consciously, many "audiophiles" fall into that trap. So, I really think it's a state of mind.
I never had a big addiction to audio equipment, though I spent at least a couple grand on amps, a sound card, headphones, etc at one point. The quality was definitely great and I could hear the differences but I honestly just couldnt justify owning that stuff considering there is always something better out there that I'd eventually want.
Nowadays I am fine with music coming out of my relatively tinny iPhone or iPad speakers, because it's just so damn affordable and convenient. When you think, something that costs $10k or < $1k can perform the same task, just "better", which one are you going to buy?