Reacting to the initial thought about very fine dining, I think that is actually something I object to much less than most eating out. Yes, going to somewhere like Alinea or Per Se or Noma is staggeringly expensive, but at that very pinnacle level, you're talking about food as art and people working at the absolute peak of their craft. I can cook quite well at home, and going to a steakhouse is pretty much a waste for me, since I can cook a nice steak at home. But I cannot come anywhere near the skill that is on display at a three star Michelin restaurant.
Of course if you're in hair on fire debt, you should not go to these places. Heck, if you're not FI, you should not go to these places. But if you have the money, and you appreciate the artistry and skill of it, I see it as something like being a patron of the fine arts, except really, really delicious.
Also, these places aren't a "rip off" in as much as they're actually not insanely profitable. They make a profit, but they have margins that are pretty typical of the restaurant industry. The price is so high largely because providing the service they are is very expensive to provide. They pay staff a lot, and they pay suppliers a lot.