I am not an expert on these, and will happily defer to some professional who's handled these transactions a few dozen times.
But here's my understanding. I think the way it works is you can do in-plan transfer of aftertax money without tax hit... as long as that money is in a separate subaccount so it's clear you're moving only aftertax money.
But, I bet you have some pretax money in the some subaccount.. I.e., you put in $10K after taxes... but between that deposit and then time you did in-plan transfer, the $10K grew to $11K. In that case, if you transfer even only just $10K, I think 10/11ths is tax free and 1/11th is taxed.
FWIW, I doubt it makes sense to convert pretax money to a Roth. Too expensive.