After 5 years I can withdraw the contribution without having reached 59.5 (of course, if it went into my taxable brokerage I could withdraw at any time without restrictions)
You don't even need to wait five years, not really.
A brief summary of the Roth IRA distribution ordering:
1) Contributions directly from your checking account to the IRA come out first.
2) Conversions from traditional retirement accounts (such as backdoor contributions) come out next, oldest first. If you converted some pre-tax money and some after-tax money in the same year, the pre-tax part comes out first.
3) Earnings within the IRA come out last.
The 10% penalty for withdrawing conversions within five years
only applies to the part that you paid tax on at the time of conversion. Remember that your principal was after-tax already. The only part that you pay tax on at conversion was any growth that happened between the time you contributed the money to your after-tax 401(k) and the time you converted it to Roth.
For example, if you contributed $10k to your after-tax 401(k) and it grew $100 larger by the time you got around to sending in the form to convert it to Roth, you could withdraw all $10,100 the next day and you would only owe a 10% early withdrawal tax on the first $100. Convert relatively soon after you contribute to the 401(k) and any taxable growth should be negligible, which should in turn mean that the penalty for withdrawing that money within five years will also be negligible. In this example we're talking about $10 of tax to withdraw $10,100. That's hardly worth worrying about.
Don't put in a mental block saying you "can't" withdraw this money when withdrawing it triggers a 0.1% tax (or whatever the real number is in your situation). Consider whether you could make good enough use of that money to make the tax worthwhile, whatever the tax rate.
Additional paperwork (it looks like I will have to mail a paper form each time I want to do a rollover)
Additional headaches at tax time, and I believe a more complicated return has a greater chance of being audited
The mega backdoor Roth doesn't make your tax return
that much more complicated at contribution time. It's just one extra line filled in on your 1040, there aren't any extra forms to file. I've been doing it since 2014 and haven't been audited yet, despite having several things that add way more complication than the mega backdoor Roth.