I rolled over two different investments from Ameriprise into Vanguard just recently, stephaniekb. I also took a hit for surrender charges. Before I did so, I figured out the difference between 3% and .05% per year. Let's walk through it. We will assume that the annuities and funds neither make nor loss money their first year. Obviously that is not a reasonable assumption, but this exercise is to isolate the the fee amounts.
Current Investment Company:
$184,000 x 3% = $5520 in fees per year
Vanguard
$184,000 - $5400 (surrender charge) = $178,600; 178,600 x .05 (VTSAX) = $89.40 in fees per year
So after one year your annuity will have $184000 - $5520 = $178,480
Investing in VTSAX with the money that you rolled over to Vanguard is $178,600 - $89.40 = 178,510.60
So, after one year you actually come out ahead, despite the surrender charges. VTSAX has a very low expense ratio. There are many funds that have higher ratios, but you get the idea.
I would double check the math. (Dammit Jim, I'm a musician not an accountant.)
This is obviously a complicated situation as forestbound and MDM point out, but I would make the change. As a matter of fact, I did make the change. My funds finished rolling over on December 1st.
Yes, it hurts to consider the amount that you have to fork over to that investment company.