If you can't get them to recategorize it and fix it at the custodian level, I'd try to work the 60 day rule here.
I'd consider the following:
Withdraw the excess money from the Roth.
Contribute the excess money to a traditional IRA as your 60 day rollover.
It seems to me like your 1099-Rs are going to be confusing and probably result in notices from the IRS - but I'm guessing you could get it fixed with some communication with the IRS. I'd probably consider paper filing this particular return (a rarity for me!) with a statement of explanation if the 1099-Rs look likely to trigger a problem.
You have 60 days to complete the rollover - so even if you make mistakes along the way, as long as you get it completed properly within 60 days, you're all good.
Even if you couldn't get it done in 60 days, I think you fall within the general safe harbor of an additional 30 days to fix an error as soon as possible.
Can you please elaborate on that, CPACat? Is it possible to roll over funds from a Roth 401 to a traditional IRA? Looking at the IRS page on the 60 day rule (https://www.irs.gov/retirement-plans/plan-participant-employee/rollovers-of-retirement-plan-and-ira-distributions) it's not clear to be exactly what kind of account can be rolled into what other kind of account (but the rollover chart on that page seems to indicate I can't), but this sentence gives me the willies: "Taxes will be withheld from a distribution from a retirement plan (see below), so you’ll have to use other funds to roll over the full amount of the distribution." My 1099-R is going to show a distribution of 100k, which will be taxable, if I roll that into a traditional IRA is that amount then deductible?
The mistake appears to be that you rolled over your deductible 401(k) funds into a Roth. So in effect, you did two rollovers:
1) After tax rollover to Roth IRA. (what you wanted to do)
2) pre-tax rollover to Roth IRA (mistake - creates tax liability and you didn't mean to do that)
You're fine with #1.
#2 is an incomplete rollover. Your money hasn't gotten to the place you want it to go - a traditional IRA. We can't reverse it and put it back into your 401(k), but you can make this a deductible-401(k) to traditional IRA rollover. So if you take it out and put it into a traditional IRA account, you complete a 401(k) to Traditional IRA rollover within 60 days. The fact that it spent a few days in the wrong account shouldn't interfere with your ability to complete the rollover into traditional IRA.
In effect, all that's happened to you is that your #2 rollover landed in the wrong account by accident. The right account is a traditional IRA account. Put it into the right account and complete your rollover.
If they withheld taxes from the deductible-to-Roth rollover, then you're right, it complicates issues - because now you have to find the money to put into the traditional IRA account.
Example:
10,000 after tax 401(k)
20,000 pre-tax 401(k)
All gets rolled over into Roth.
They withhold $5000 for taxes - so only $25,000 lands in your Roth.
You take out $15,000 and add $5,000 of your money to replace the withholding and put $20,000 into a traditional IRA. You have now have of withdrawal of $20,000 from your pre-tax 401(k) which you rolled over into a traditional IRA. The withholding comes back to you when you file taxes.