I wonder if there is an economy of scale effect at all with regard to the 20% "pink" tax.
Men, in general, tend to be a bit boring with a lot of fashion. Clothes come in three colors: blue, black, or grey. We tend to buy lots of tshirts and jeans, and are upset when the Levi's cut of "straight leg, comfort fit" jeans changes for the first time in 30 years to reflect updated styles. This also carries over to dress shirts, suits, shoes, etc.
Could it be that men's clothes are more rugged and cheaper because they can produce a lot more of the same and no one cares? Conversely (mustachian women aside), women's clothing lines tend to be shorter, smaller runs of product due to fashion changes, and tend to encompass a wider variety of colors and styles.