The thing that I've found difficult is pregnancy and childbirth.
My body shifted after each kid.
So, I was fat at one point (5'2.5", 182 lbs).
Then I lost weight, got down to 125-127
Then I had a baby
I did manage to get to 127 after him (eventually), but it was super hard to stay there, and 132 was more realistic.
Even at 127, things didn't quite fit the same way.
Then I had a second baby when I was 42.
It took 1.5 years before I could lose much of the weight. So I was at 168 pounds for a long time. And bought clothes for that. But I held on to my smaller clothes.
Now I'm 140-145. Been trying to lose those last 5 pounds (so a goal range of 135-140) since December.
I dug out my clothes. So my closet is packed, mind you, with stuff from an 8 to a 14.
Finally let go of the 14s, they are too big.
Some of the casual stuff that I had before the 2nd baby - you know, even 5 lbs isn't going to cut it. I let those go too.
Turns out that without realizing it, most of my work clothing (jeans, casual pants) started fitting - probably in November but didn't realize it until Feb.
The cute shirts - well, you know, my shoulders and ribcage got broader with the second baby. I don't think they are ever going to fit. So I let them go too.
In some cases "just a few pounds" makes things fit better.
In some cases "just a few pounds" is a crap-ton of work. If it takes perfection 95% of the time? It's not sustainable.
In some cases "just a few pounds" won't shift where the weight lies.
I know, for example, that my weight fluctuates by 5 lbs easy over the month.
Some of my older clothing, that I let go? Really only fits 2 days a month.
So where does your weight come from, and how do your clothes fit? Me, it comes in the waist and butt, and 5 lbs makes a big difference. For my husband? Not so much, the tummy can hang over the belt a little more.
I no longer aim for tight-fitting clothing that is not forgiving (never really was into that) - you know, like skinny jeans and such.