Really there is a pretty simple answer for everyone, regardless of the size of the mortgage to at least get you started:
- Do the legwork and find out what it would cost to refinance and how much you'd save in monthly payments.
- Divide the amount it costs in total by the amount you save each month, to determine how many months it'll take until you break even.
If you can't guarantee you'll be in the house in that amount of time, then it's not worth refinancing.
Example: Say it costs $4,000 to refinance and you save $75/month, well then that's 53+ months (4 years, 5 months) to break even. That gives you a better idea if it's worth it.
Obviously there are other variables, such as I'm 5 years in and so have 25 years left, but I'd move to a 30-year payment, etc. But the method above will give you a much better answer than any of us can.