Actually you can just buy VTSMX. It's the same as VTSAX, but with a little higher expense ratio. However, at which point that you reach $10,000 in total invested in VTSMX, it will actually get upgraded to VTSAX shares automatically.
You can pretty much put the whole $5500 into the fund, rather than worry about rounding issues with VTI.
You cannot own shares across accounts, though. So it does seem like, for now, the $5500 in your tIRA will remain VTSMX, and initially the $5500 in your new Roth IRA will be VTSMX. (I do not think Vanguard "adds up" the collective VTSMX shares and upgrades you.)