I probably would not give up easy $ if it was low stress, but I also gain some enjoyment out of succeeding and doing a job well. But if I had more money than I'd ever need and thought I could do something better with the time? Yes I'd give it up.
Is there something you'd rather do? Or if you are working and making good money, could your purpose be growing your wealth in order to make larger and larger contributions to charity? It really just depends want you want in life. I'm about a decade away from FIRE still, but I know that I do not want to be just all about myself when I RE. I plan to volunteer, or if I'm FI, keep working and make huge donations to my church and to caring for the poor and abused in my city.
A friend of mine says he could never RE because he "doesn't want to leave money on the table". He has a savings rate higher than mine, I really have no idea what he'll ever use the money for. But he likes to see his balances grow. That's his reason to keep working (I admit I like seeing mine grow too).