We did both. Moved to Phoenix for a big bump in pay (for me, and eventually for DH). Lived there for 3 years. Moved back to CO for about a $20K/yr drop in pay. If I had it to do over, never would have moved in the first place. Would have moved back sooner if we could have. Due to the housing crisis and timing, we wound up losing all the gains in income anyway, but had we stayed we'd have been closer to ER for sure. Even though my job now is just OK (not the suckiest ever, but not what I was "born to do") I'd rather work 3 more years here than have lived 3 more months there.
If ER is fairly close, you will have enough $ to move to a place where you'll thrive after becoming FI then it's likely worth it to stick it out. If you cannot figure out a workable plan to eventually move to a place where you're happier after ER, you should revisit your plans. Such as "on this track, we'll be ready to move to Desired Location and be FI at age 40". If you're 37, might be worth it if you can stand it. If you're 27 you might think about a plan B.
Some criteria might also be in order:
-your health or mental health. If living somewhere is seriously impacting your health, consider moving.
-being near people who might not realistically make it til you move to desired location. You can't get back time with Grandma or Dad or having your kids know Great-Grandpa or Aunt Jane. Being closer to family might be worth extra time working, or extra time in not your ideal place.
-realism about how great desired location will be. I know people who moved to Boulder because they thought: Boulder is *awesome*!!! But there are some things they found they dind't consider and didn't like about it (lack of diversity, the general attitude of people didn't fit them, the weather was more unpredictable than they wanted, not much to do if you're not into the outdoors or beer, etc). Every place has drawbacks and sometimes a blessing is also a curse. Hard to imagine getting sick of the sun when you're in the midwest or PNW in winter, but some people who move to the desert SW or SoCal eventually find they get sick of the sun or the lack of seasonal variation.