Fees. This actually hits you 2 ways:
1. Vanguard Funds are Transaction-Fee funds at Fidelity. So every time you buy or sell, you're paying $49.95, if I'm reading the fine print right.
2. Investor-class funds only - no admiral shares. So your annual expense from the fund is higher. Example - VFINX is an S&P 500 fund you can buy at Fidelity. It has a 0.16% expense ratio. At Vanguard, once you have $10K in the fund, you can get Admiral Shares. Same underlying investments, but a lower fee - 0.05%.
If you want to stay at Fidelity, use their Spartan Funds. If you want Vanguard funds, cheapest to move your money to Vanguard. If you want both, have an account at each.