Not much advice here, but DH struggles with this, except that he makes ~$10 an hour at a coffee shop. He wants the shop to do well but a lot of the staff there don't take the job seriously. He takes pride in his work and can't half-ass things. It gets very frustrating for him that he cares so much to improve the place, but he often wishes he were just another lemming. So if you think you want to quit your higher paying office job and just work at a simple job like food service... the grass might not be greener than it appears.
On the flip side, I do purposefully avoid any managing responsibilities at my job. My career has spanned 14 years now and I've been a senior accountant for 12 of them. In my very first senior accountant role, I did manage a small staff, and it was okay, but I just like the ease of what I'm doing now. It's so easy, in fact, that come FI, it will be hard to leave it for RE. Probably will restructure my schedule, more telecommuting, part-time work, etc.