Some brokerage accounts only let you buy Vanguard ETFs and not mutual funds.
Downsides to IB that I’ve noticed: very confusing interface, requires a Java app just to show cost basis, obnoxious upsell emails that are hard to opt out of, and many more.
Downsides to M1 I’ve read about (I don’t have an account there yet): no limit orders on ETFs, can only execute orders at two specific times each day (or one time if you don’t pay for M1 Plus).
General downsides to other brokerages: sometimes there are fees for mutual funds or you’re limited to ETFs only.