I fly fish... it is nowhere near as expensive as folks make it out to be. Orvis offers free "learn to fly fish" classes, though they try to get you to buy their VERY expensive equipment. Craigslist is a great source for cheap, good, used equipment. There are also a number of websites/chat rooms/forums where you can pick up equipment on the cheap.
Start Up Costs:
Rod: $30
Reel (with line): $25
Tipped: $5
Flies: $10 (then learn to tie one or two some patterns)
Licence: $25? (Varies by state)
So under $100 gets you a VERY nice fishing set up.
When I travel, I stop in at the local fly shop, ask what flies they recommend for the local water, and buy some. Shop owners are much more likely to point you towards some good water if you drop $10-$20 on a couple of flies. With guided trips running $100+ a day, I find the purchase of flies to be a bargain.
To get a better handle on your local water, join your local chapter of Trout Unlimited. Remember that free Orvis class I recommended? One of the perks is a free one year membership. A good local chapter will have members that are willing to help you learn... usually for the bargain price of a domestic six-pack.
Like most hobbies, fly fishing CAN get very expensive, but hell... so can running. Have fun with it!