Golf is like a house, you can make the experience as frugal or as crazy expensive as you want it to be. Public golf on the cheap isn't bad, I've seen crazy good deals at golfnow.com for a few of the local courses. Can you barter at a nearby club (act as a ranger, starter, provide them a service they would otherwise pay for)? I still fix some computers on the side for my former country club and manage to play there about once a month.
The exercise isn't bad at all if you walk, and given I am of a certain age walking five miles or so is a decent amount of effort.
Keep the clubs if you can lower the costs.
Alternately, disc golf is quite fun.