If you are trying to reach FI faster, then yeah, it's silly not to consider it. But if it's more likely to result in you working just as long, but just amping up your spending, then it's not necessarily worth the risk.
The risk is relative though. You like your job, but how rare is the kind of job you have? Do you like it just because you have a good team and the work is interesting? Or do you like it because this is the one company that allows you to work the way you've always wanted to?
I turned down many much higher paying jobs while I was working because no one could offer what I had, most workplaces in my industry were miserable by comparison, and I was in no rush to stop working.
DH on the other hand works in an industry where he made 4 quick lateral moves over the course of 18 months because he knew it was that easy to find a really great fit on a really great project, so he just hopped around until he landed somewhere he was happy.
For me to turn down huge money offers made perfect sense. Losing what I had meant perhaps permanently damaging my ability to enjoy my career. For DH to pass up huge money just because he likes his current job would be insane, because it's the easiest thing in the world for him to find another great position.
So no one here can really answer this for you, it's entirely personal and depends on what you want your career to do for you.