Unless they change their policy, you (or your heirs) are going to get dinged by that $50 fee whether you close it now or later. And in the meantime, they are probably charging you 1% expense ratios for your funds instead of the 0.05% expense ratio you could get at Vanguard. If so, you would save that $50 in under a year by switching. Just do it.