I hear you and offer you a virtual bottle of Angry Orchard Apple Cider.
You are bucking up against The Costume.
As if, somehow, the clothes make a difference in your expertise, knowledge, or credibility.
I know my stuff. Whether i am kicking back in sweat pants or wearing the suit, it doesn't matter.
To be honest I would probably work for fun if it didn't involve the stupid costume that is out of sync with Texas weather (oh, plus the insane hours). My clothes turn into something sinister around May - it's inhumane. Why? Why should I wear a suit in the Texas summer so that some shallow customer thinks me an expert? Dammit, I know my stuff, already.
FWIW...
If you pick a neutral color for the main stuff you can eventually build a mix/match wardrobe with second hand stuff that is cheap and good, even if you are an unusual size.
I go with a similar color scheme for everything...black. I'm female, so that means black skirts and jackets (and shoes). These aren't always easy to find at thrift stores - but persistence can really pay back. I add a different color shirt/accessory as needed.
My boss has mentioned several times that I wear a lot of black. My comeback is that "my skin tone" works best with "jewel tones" and these need a lot of black to "pop" when I do presentations. YMMV.
My boss is the only one who has mentioned it - I don't see it as a big deal, because I KNOW my stuff. If you are middle of the pack in your field, you might need to work harder on the the costuming until you are established.
/rant.