I currently owe $75,000 on the loan (initially $138,000). My plan was not to refinance because I was hoping to have the mortgage paid in the next couple of years and I really don't want to mess with getting an appraisal, etc. However, if I can find a program where it makes sense, then I don't see any problem in refinancing.
Also, where are the fees built in? He said it wouldn't cost any money to lower to 4.5%. He advised if I wanted to go to 4.375% then it would cost $400.