I think it works like this:
I had $3000 in the IRA, I added $5500 more. So I divide the new contribution ($5500) by the total amount in the IRA ($8500) to get the percentage that I can convert tax free, which is 65%. 65% of the $5500 is $3575, which means the remainder ($1925) will be counted as taxable income, taxed at the highest marginal rate. If I moved all $8500 into the Roth I would be adding $3000 to my taxable income. I'm not sure why I'd want to do that, since I will be taxed less in retirement than I am now, but maybe I'm missing something?
I would be on the hook for the whole $3000 because I've never paid taxes on it (they were deductible contributions).
If I'm able to just move the $3000 into my 401k without any more steps that would be pretty easy. What you said makes sense, if the cost basis is determined by the combined balance of my IRAs at the end of the year, then converting to 401k now should work. I guess I was sort of worried about what they would count as the end of the year.