There was a rather lengthy discussion awhile back (sorry i don't have a link) about the real effect of confidence on others opinion of you, irrespective of what you wear. As long as you meet the social norm of what you are expected to be wearing (i.e. not wearing t-shirt and sweatpants to a business casual atmosphere) people perceive your confidence level much more than the actual clothes you are wearing...unless of course you are trying to schmooze with some snobs, but why would you want to be friends with them anyway ;-)
This more or less matches my experience: intellectually, it makes some sense that appearance could matter, and I could see that appearance might serve as a sort of substitute for confidence in many circumstances, but I don't personally attend to my appearance at all, other than maintaining very very basic professional standards - and by that I mean far less attention to those standards than most women would, and it's just never, ever been an issue. I am short, really emphatically not physically attractive, don't wear make-up or do anything else to try to conceal that basic fact, wouldn't have a clue what's fashionable, and am so bored by shopping that something else is /always/ higher on the priority list, until the point that my clothing begins to wear out, possibly beyond the degree that it's really okay in a professional space - but I've always been a high performer, I can command a room, and I am known for often being able to solve other people's random professional problems quickly in the course of drive-by interactions: I assume that I'm useful enough that, if people are having bad reactions to how I look, they must suppress them enough that I don't sense it. But I also suspect that my own total lack of care about this kind of stuff is sort of infectious - other people, in professional spaces at least, take their cues from me, and if I'm not bothered, neither are they.
That said: I don't do anything disrespectful: if I'm in a context that requires a suit, I'll wear one, etc. But, really, it would be difficult for me to do less to attend to how I look.
At the same time, I do believe this /can/ matter, so I don't discount as a real constraint on what other people can do. I produced some publicly available video content recently for one of my courses, and I wondered briefly whether my appearance would draw drive-by criticisms online, because that kind of random negativity happens so commonly and it's so easy for people to do anonymously. It hasn't happened yet, but the fact that I thought about it means that even I am not completely oblivious to the constraints. It wouldn't take much for that awareness to dent someone's own confidence, which could then create a cycle where they are less able to behave in ways that deflect other people's attention from appearance. And of course there are types of roles where the pressure will be more intense, objectively, as well as structural situations where someone is too disempowered for their own self-confidence to matter as much... In many circumstances, it may be worth attending to the social standard, at least to some degree, because you need to pick your battles, and this may not be the best one to take on for a particular person...
These comments are all with reference to /professional/ spaces. If we're just talking private social circles: I would change my social group rather than have to worry about this kind of thing in any private space. That should be a realm of freedom from this sort of constraint.