If you extrapolate from your thought that you have something in your job that another would work very hard for a chance at, then leaving because you're FI could potentially open up that opportunity to this 'other'. The company would still need someone to carry out your duties so would this not remove the 'guilt' part of the equation by allowing you to provide this opportunity to another hard-working soul?
Yeah this!
As a Scientist, I think this will be a huge thing for me when I eventually retire (not that I have even 1% of the same type of guilt as the original poster). In Science/Academia, I guess a huge part of people's motivation is to make the world a better place. Yes, I can make the world a better place by staying in the job, but so could a bright and capable young Scientist with huge potential and new ideas! The way things work in this area, most get to finish their PhDs, and then there is no opportunity left for them, so they leave Research/Academia. By leaving, I'll open up an opportunity. And its not like I won't still get to try make the world a better place by myself. Its pretty easy to volunteer yourself to a University after you retire at any age and contribute to your field almost as much as you want. So FIRE for me might essentially be me getting to do the things I love the most as much as I feel like (eg working with postgrads and mentoring, coming up with new ideas, working on really cool things that are important but for some reason society doesn't pay for), without having to do the stuff I don't want to do (teaching courses I don't like, admin duties, working on projects just because they bring in funding).
So, OP, maybe think this way. Think about, who is a younger person who would love your opportunity, that you could fill your role with? Prior to your FIRE date, spend time working on potential candidates. Mentor them. Prepare them. Give back in this way :-) Then when you FIRE, if you feel guilt about not working, find ways to contribute to Society that you like. If you are in Law (someone mentioned?) there are loads of ways to do this! I'm sure you can think of them straight away :-)
FIRE is an avenue to work on the things that really inspire you and perhaps drew you to your field in the first place, without having to sell yourself out any more :-)