Be careful, the lowest fee fund isn't always best. Take Fidelity ZERO funds for example, they're free! Except that they intentionally hold excess which creates a drag on performance, and net of fees they underperform Vanguard. Vanguard isn't a for-profit company like Fidelity or Schwab, they're a mutual. If low-cost indexing is your thing, you can't go wrong with Vanguard.
I hold many "total US stock" funds and ETFs. Sorry, but thus far, FZROX has edged out all of my other investments. By pennies, but the statement that they're somehow dragging isn't correct. Time periods are going to matter. I watch both of my Fidelity funds, FZROX and FSKAX along with SCHB, VTI and SPTM. The differences between any of them is a big "who cares?".
To the OP, I would not have any problems with swaps between good Vanguard and good Fidelity funds. Be happy you're not stuck with Mass Mutual or NY Life funds.
$10,000 invested in FZROX on 8/2/2018 when the product was created has turned into $10,625.75. Same investment into VTI is now $10,696.31. Investing with Vanguard would have made you an extra $70.56 in less than a year. $70 on a $10,000 investment is 70 basis points. If you project earning somewhere around 6%/year in retirement, an extra 0.70% will have a significant difference on outcomes.
The difference is the cash holding, this gap will only grow over time.
*EDIT - my apologies, I'm wrong on this one. The data above is accurate, but the majority of the outperformance occurred on the first day. Something to do with Fidelity's trading on the first day I'm sure. FZROX only has 0.29% in cash, and has performed relatively in-line with VTI after that first day, $8 of outperformance for Vanguard. Watch out though, they aren't all the same. FZILX, the international version, has 6.07% in cash.