I really enjoy the occasional coffee shop visit (latte, book, 2 uninterrupted hours someplace where neither children nor work can reach me... bliss, and totally worth the 5$ for the latte).
That said, I also find that I enjoy it a lot more when it's an infrequent thing. It's so easy to slip into a habit of spending 5$ for every cup of coffee, especially when everyone else does it, and then it's no longer a luxury, it's just a part of the routine. A routine that, as you pointed out, costs upwards of 1K/year. Eek. (Similarly expensive routines: lunches out, drinks out on a regular basis, take-out meals...)
Personally, I've found that a latte out every two months or so feels plenty luxurious, but doesn't derail finances in a significant way. I think the balance is important - NEVER doing it would feel like deprivation. a 2$/month budget doesn't feel extravagant, but MAN is that alone time ever good!