DON'T DO IT!!!!
I did just that and I have been regretting it every day. The only benefit that I have seen is that Vanguard doesn't have a minimum hold of 1 month for free trades of their etfs and funds. Otherwise, Fidelity has slightly cheaper funds and etfs. Their sector etfs are 0.12% vs 0.14% for Vanguard, and there are some other funds that are relatively cheaper.
Also, Fidelity's website is much easier to use. Vanguard has a weird 2 account setup for each account you have. One is a Vanguard fund account and the other is a brokerage account. It's confusing and silly. Also, trading is confusing on their website.
Most frustrating of all are the Vanguard customer service reps. I have been on the phone with them over dozen times for various problems from bank transfer, to account set up, etc. I don't know why but all of them seem slightly stoopid or just slightly rude. Fidelity's reps always seem more polished and friendlier and always get the issues resolved for me. Vanguard's reps always seem angry or bored or just negative. They also put you on obscenely long holds with no music so after 10 min, I don't know if I'm still on hold or not.
Anyhow, I'd go back to Fidelity except that my company has it's IRA with them and my parents now, thanks to me... :( and so on so it's a big hassle to switch back.
But if you can choose to stay with Fidelity, do it!