If you're planning to stay in that home more than a couple years, then refinancing makes sense. You'd be saving about $2k per year on interest.
If you're concerned about losing 3 years of paydown, you're right--when all is said and done, you're basically starting your mortgage over. You're currently paying off about $370/mo of interest, and you've paid off maybe $13k total since you started. You're giving up about half that ($5k ish) in closing costs and such. If you're saving $2k in interest per year, though, you'll make that back in 2.5 years, and then continue to save $2k per year (after taxes! woohoo!) from then on.