I think if you're feeling socially uncomfortable in your clothing, that's as bad as feeling physically uncomfortable. Sporting a formerly trendy pattern that is now classified "butt ugly" can socially be as bad as wearing a giant coffee stain. Rips and stains are the obvious signs of wear, but fading, stretching, and pilling are more gradual and thus harder to notice. So is becoming "dated". No one is going to tell you "ugh, that cardigan/jacket/pants/etc is so last year"...it's that the overall impression you make will become "she's frumpy/sloppy" or "that outfit ages her/is unflattering" etc. As someone pointed out, it's not too expensive to upgrade a small wardrobe a little at a time, perhaps thrifting or using discount stores or shopping good sales.
I absolutely don't follow the whims of fashion, but I do try to be up to date in professional settings; while it may not matter what people think of you while you're at home or running errands or outdoors, in some settings the impression you make matters. Your appearance directly affects people's perception of your competence, your "fit", the likelihood someone will strike up casual conversation. Appearance isn't just wardrobe, and people who are innately more pretty/handsome, or more fit, can get away with more in wardrobe, but you can't change your face (at least not frugally), and changing your fitness level is a slow road, and you have to wear clothes and move along in your career when you're on it!
By the way, in addition to fashion being more finicky (more embellishments, more variations even in simple pieces), and using thinner fabrics that wear out easier, women's fashion is difficult because women vary so much...Button down shirts are a classic but I had a lot of trouble finding one that didn't gape in the front or fit like a tent. So...sometimes we have to deviate from current trends and just wear what fits and flatters unapologetically, and call that "timeless style" ;)
Clothes that aren't top notch anymore can be downgraded to everyday or home clothes, then (sometimes) to dirty-task clothes. If they just don't fit, or your taste has changed but it's still in decent shape, giving it away or donating it keeps it out of the landfill. Perhaps someone will pull a "refashionista" on the dated piece, or just use it as raw material. Worst case, thrift stores route anything they can't sell into recycling.