You only pay zero tax on Roth conversions to the extent that they are less than the standard deduction and personal exemption ($10k single/$20k married). In your example ($260k pre-tax growing at 8% annually), the earnings in the account each year ($20.8k) are more than enough to max out that space for a married couple. If you leave your $170k of after-tax earnings in a tax-deferred status, that amount will accumulate another $13.6k of tax-deferred earnings each year. So you either have to withdraw more than the earnings each year and pay some tax or stick to the 0% bracket and just let your pre-tax money grow indefinitely. In this case, you would have been better off putting your post-tax funds in a taxable account because you really would pay 0% tax on this $13.6k during retirement.
Having money in a traditional after-tax 401(k) during retirement is almost never a better idea than keeping that money in a taxable account. If you have the after-tax option available, it should only be used as a way to stuff your Roth with more money than you would otherwise be able to contribute. If you're worried about having access to enough money during pre-59˝ retirement, use a taxable account instead so that the earnings are taxed at a lower rate when you sell.
The earnings in the pre-tax account would not outpace the available space for non-taxable Roth conversions, because the standard deduction and personal exemption are increased each year at a rate approximating the inflation rate. (Your numbers incorrectly assume that the annual amount that can be converted tax-free in the Roth pipeline remains constant, unadjusted for inflation, for thirty years.) Cfiresim tells me that historically, during those thirty years this hypothetical married couple would completely exhaust the $260k pre-tax account 83 out of 115 times if we continue to assume a constant 8% return (and 85 out of 115 times if we use actual historical performance) (in either case, using cfiresim's default settings for the other variables, including a 75/25 stock/bond allocation). This couple is going to need to access the $170k of after-tax earnings in tax-deferred status during those thirty years too.
Are we using inflation in this scenario or not? I figured we were ignoring it because of the assumption of constant $17.5k/$34.5k contributions to the 401(k) (these limits are indexed to inflation as well). But if we're ignoring inflation, 8% return on investments is a bit higher than I would assume, so I concede that the earnings on the pre-tax account would probably not actually be higher than the 0% tax bracket most years for a married couple.
I will also concede that the couple in your example would probably be better off leaving their money in their after-tax 401(k) than converting to Roth. They would even be better off doing this than having the money in a taxable account because they wouldn't need to pay tax on dividends/capital gains during their working years.
However, I would also argue that this is pretty much the most extreme possible scenario.
If the worker had earned any more each year, they would have had some money left over to save in a taxable account, increasing the fraction of their money that isn't earnings on after-tax contributions.
If the worker had earned any less each year, they would have been unable to max out the after-tax 401(k). This would have also increased the fraction of their money that isn't earnings on after-tax contributions.
If the worker had an employer match in their 401(k), they would have more pre-tax money going in there each year, which would have reduced their after-tax contribution limit, which would have also increased the fraction of their money that isn't earnings on after-tax contributions.
If the worker's spouse also worked for a company offering a 401(k), each earning $50k instead of one earning $100k, the couple would have doubled their pre-tax contributions and the after-tax contributions would have been cut roughly in half, which would once again increase the fraction of their money that isn't earnings on after-tax contributions.
Basically any variable you tweak in your scenario increases the probability that the couple would be able to make it to age 59˝ without touching the earnings on their after-tax contributions. There may be many real-life couples who are similar enough to your hypothetical couple that they would benefit from making after-tax 401(k) contributions and not converting them to Roth right away. However, I stand by my statement that this is a sub-optimal strategy for the majority of early retirees. Most would be better off either saving in a taxable account or making after-tax 401(k) contributions and regularly converting them to Roth.