I think that might speak a bit to your concerns about the general math, and Contingency E. Of course, if you can't reproduce the numbers in a spread sheet, then it's certainly possible I did something wrong. I get the sense that you haven't gone that far yet, so I'm just trying to explain how they came out that way for me.
Did you factor in the LTCG rates for redemption in a taxable account? I ran some numbers, with some help of Bogleheads formulas
http://www.bogleheads.org/forum/viewtopic.php?f=10&t=140669Assumptions:
Start with $10k, only looking at using VTSAX in a taxable account vs a 529 (I'm assuming you're putting your tax inefficient funds elsewhere).
2.1% dividend yield for VTSAX
For the sake of the argument for tax deferral, assume federal 43.4% marginal income tax rate and 23.8% LTCG rate (the extra 3.8% because of ACA).
I will assume GA state tax rates, because I am familiar with those. 6% marginal rate on income over $7k after deductions.
Use a NY 529. I only checked Vanguard (I think that's NV), Utah, and NY 529s because they are the ones highest regarded on Bogleheads forum. There may exist a 529 with a US domestic stock fund with a better ER, but honestly, the ER effect isn't that big.
In the taxable account, the tax drag is equivalent to an extra 0.650% ER. VTSAX has a ER of .05%, so for comparison to a 529, it's got an ER of 0.7%.
In the NY 529, the ER of the total US stock fund is 0.16%.
After 20 years, the balances are 40925 in the taxable, and 45247 in the 529.
At redemption, in the taxable account, assume that all of the capital gains are long term capital gains. Capital gains tax is assessed on $30925
The penalty on the 529 is $3524. But state taxes do not have a long term capital gains rate.
Various redemption scenarios:
1) Assume that all of the money at 15% marginal / 0% capital gains rate + 6% state. This assumes that the retiree has other sources of taxable income that pushes him or her up to the 15%/0% bracket.
Tax + penalty on 529 gains: .21*(35247)+3524 = 10925.
Leftover after redemption: 34322
Tax on taxable account gains: .06(35247) = 2114
Leftover after redemption: 38811
2) Assume that all the money is redeemed at 10% marginal / 0% capital gains rate. Again, assume the retiree has other sources of taxable income that pushes him up to the 10%/0% bracket + 6% state
Tax + penalty on 529 gains: .16*(35247)+3524 = 9163
Leftover after redemption: 36084
Tax on taxable account gains: .06(35247) = 2114
Leftover after redemption: 38811
Those two scenarios above have the maximum tax deferral effect - highest tax bracket during investment, and low tax brackets at redemption.
If I rerun the numbers over 40 years:
529 balance: 204736. Penalty: 19474
Leftover at 15% + 6% state = 144367.
Leftover at 10% + 6% state = 154104.
Taxable balance: 167489.
Leftover at 0% LTCG + 6% state = 158040
Perhaps I made a math error, but I can't seem to find a scenario where you can come out ahead with a penalty.
EDIT: I assumed that AMT doesn't come into play, because I have no understanding of how it works, other than it sucks.
Note also that the core claim isn't about winning by paying a penalty. (Though that's possible far enough out, see E.) I said that if I had $10k too little in my 529, then that would cost me 4x as much (in taxes) over the long haul as having $10k too little (in penalties). It's ideal to save exactly the right amount in your 529, but the odds of that happening are tiny. Our intuition is to aim for the true target as best we can, but I think that's wrong: an asymmetry in costs means you should lead your target a bit, and be content with the understanding that that means you intend to oversave most of the time as a hedge against the times when you would otherwise have undersaved. So having a quantitative view of the relative costs of the two options is really helpful. Note also that for someone in a lower low tax bracket, the ratio would be very different. This is all very individual.
Could you explain more by your distinction between winning despite a penalty vs "I said that if I had $10k too little in my 529, then that would cost me 4x as much (in taxes) over the long haul as having $10k too little (in penalties)"