People don't like what they don't like, that's fine, but golf does not have to be an expensive hobby at all. Plenty of good golf to be played for <$20/round on public courses.
With what clubs?
You're right that golf doesn't have to be as expensive as some people make it, but if you contrast it with soccer, basketball or tennis, which can often be played for free in public parks with less expensive equipment, golf is more costly.
Used clubs. I have a total of $300 into my golf clubs. More than half of that is in my two putters--but I can get a superb putter for $10, and I'm very unusual in carrying 2. Actually, a full 1/2 of that is in one putter.
I could buy, and have bought, a set of excellent golf clubs including bag, driver, putter, wedges, and irons, for less than $75--and unless you play 100 rounds a year, your clubs will usually last 10-30 years, unless your swing speed changes dramatically in that time.
I'll say that my golf addiction is cheaper than my hockey, racing, and fishing addictions. The only reason my Guitar addiction is cheaper is because they are legitimately appreciating investments.