I really like the idea of "failing fast," but I admit that it's hard to put full effort into succeeding/failing quickly when it's a side gig. I am in a similar situation, having started a side business; it's meagerly successful, but probably not enough to justify the time I put towards it. I think if I had an extra 10-20 hours to put towards it every week, I could prove it a success or failure within 6 months. Instead, I am left dinking around on it when I have time, and hoping for success in my limited time.
Specifically about your business: pricing is probably the hardest thing to get right. You've purchased inventory, so you've calculated what you need to make on each item to make it worthwhile - there's probably not a lot of flexibility in how much you charge. Therefore, you have to find the right audience... which is the incredibly difficult part of any business. Today's modern advertising options are pretty good at targeting based on demographics and interest, so if you haven't already tried, maybe try some twitter/facebook/adwords ads that target a very specific audience? Do you have competition that you can analyze and emulate? Can you find a few men's fashion related websites and buy direct advertising?
I work in web marketing, and I've noticed that new websites rarely get any search engine traction in their first year, so if you're able to hold tight until 1+ years online, you may start to see some organic search engine results (assuming you have somewhat unique products and aren't competing with the big guys).
If your current efforts are already bringing in a few interested website users, maybe take another look at the website itself? Does it appear trustworthy? If it wasn't your business, would you buy from yourself? Is there anything you can do to add trust? (secure certificate, add PayPal as a payment option, sell through amazon/ebay as well?). If the user doesn't feel 100% confident ordering, they probably won't.
