Don't feel badly. We all make mistakes.
For the annuity that is currently at $138K, yes, I would absolutely pay the penalty and get it out. Here are the numbers: If you subtract $5K from the current balance, you have $133K. Even if you only make 5% on that for 6 years, you should end up with around $178K. I'm not sure what rate of return the annuity offers, but you can run the numbers on that and see where you stand. More than anything, though, it sounds like it's bugging you and you don't trust the company, and that right there would be enough to make me feel like I needed to get the money out and roll it into the Vanguard IRA.
I'm not so sure about the other situation, but I think you'd have to run some numbers and see at what point it makes the most sense to roll it over. Or, for the sake of peace of mind, you could just bite the bullet and do it now, regardless of what the numbers show. (That's probably what I would do, but I place a huge value on trusting the companies that hold my money.)