I think paying back the loan with after-tax dollars is the kicker.
Saying the TSP grows faster needs to be parsed out. The TSP and the taxable accounts, as distinct stand-alone accounts, will grow at the same rate. However, I'll have to pay some taxes each year on the dividends in the taxable account (but on $50K, that's not that much), which on a ledger, you could deduct from the taxable account's value, so in that sense, it's not "growing" as fast. But that is not a big difference.
I'm not sure how you calculated the marginal rate to calculate the cost of the loan. Whatever the cost, however, it all goes to me since it's a loan to myself (and the interest is approximately 2%), right? Also, the interest/gains from the loan are not presently taxable since the loan is inside of the TSP. So, I don't know how that's an issue. Yes, I'll pay taxes on the TSP withdrawals later, but that's true however I earn the money.