Pretty much all your options are:
-straight from the tap
-cheap/simple filter from the tap (pitcher, on-tap contraption)
-more $$ filter/purifier under the sink
-bulk purchase (reusable container)
-bulk purchase (disposable container)
-individually sized purchase (disposable container)
What will work for you depends on your tastes, your water source/flavor, your other values or worries, and your housing (space for storage? liberty to install RO system?). Obviously any disposable-container options are shamefully wasteful, last-resort options. Likewise, drinking straight from the tap is the most frugal and sustainable. But I, while I drink freely from the tap in Chicago and rural Iowa and San Jose, I cannot stomach water in Los Angeles/SoCal and especially not in Vegas; I swear the SODA from fountain machines smells and tastes like chlorine and iodine. I can barely brush my teeth in Vegas.
So: price out your options (accounting for initial and running costs like new filters), figure out how you value non-cost externalities (gas to drive to the store vs waste from filters; though if you get a SOMA filter those are biodegradable!) then decide if trying a filter is worth the investment if you are happy with your 25c/gal option. Assuming you REALLY love to hydrate and actually drink 64 oz/day of straight water (unnecessary, since much of our intake is from food), that's a whopping 3.75/mo/person. Clearly, it will bankrupt you very quickly.
If you were showering in that water I'd see it differently, but it's a matter of scale. Do you really have your expenses optimized to the point of 3.75/mo/person?
One last tip: if your tap water is borderline and it's really just the flavor that concerns you (not safety, etc), get a nice pitcher and slice up a cucumber or a lime in there, fill it up, and throw in the fridge for a day or so. Herbs from your windowsill also work great; in fact you can use any fruit/veg/herb but it stops making sense for water treatment if it's not cheap...