I honestly would not shift from Vanguard to Fido. And I am a really big fan of Fido and have all my portfolio in with them.
There is nothing wrong with just starting a new set of accounts with Fido and keeping the Vanguard accounts "as is." You don't have to have everything all in one spot. Not that there is anything wrong with doing so, as Fido and Vanguard are both stable, reliable companies I'd have no issues with, but cashing out a taxable account (and incurring expenses to do so) just to move it over from one to the other is just... not necessary. They're both perfectly fine.
If you use a budget program like Mint or Personal Capital, you can still see all of your accounts in one place for purposes of monitoring.