Forgive me if I'm wrong, but it sounds like you might be under the misapprehension that you can't access retirement accounts during early retirement. See if this blog post clears things up for you.
I'm aware that you can and aware of the concept of roth conversions, but I do know they have income, timing, and tax considerations that make it situational. There's also the option to early withdrawl at a penalty from pre-tax accounts, which again, has it's downsides. I was more curious if the downsides and decreased flexibility of those options outweigh the tax disadvantages of post-tax brokerage account investments. Again, my goal is flexibility and lack of financial stress so I'm willing to sacrifice some gains for convenience and flexibility.
I find having to track my trades and capital gains from my post-tax brokerage account financially stressful. It's better now that brokerages are required to track and report your cost basis, but I'd rather not have to do it.
The disadvantage of taxable accounts depends somewhat on your behavior in it too. Every time you sell at a profit you incur a tax in your taxable account, vs only when you withdrawal funds in an IRA. So every time you rebalance you'll incur taxes. Also, even if you just hold the S&P 500 and never sell, you'll have taxes due to dividends and the fund moving out of companies that drop off of the S&P 500.
I've had some really high gains in my taxable account due to holding options that increased substantially. The thing with options is they have expiration dates, so even if you want to continue holding the company using options, you have to sell ones that are expiring to buy ones that haven't expired yet. In an IRA, these gains aren't an issue because the sale isn't a taxable event.
I also funded a rental house purchase, and repairs from my taxable account. I didn't have to have any penalties, but I did need to note down my capital gains and include them on my taxes. When I refinanced the rental house years later, there were no restrictions on how much I could put into my taxable account. I'd call these benefits.