I did a partial IRA to Roth conversion a few years ago, so maybe I can explain how it works. This is all from memory so it's possible I got a detail or two wrong, but the key takeaway is that it's easy.
To actually execute the transaction with Vanguard, all you do is use their "Exchange" interface that you'd use any time you wanted to rebalance. You sell some assets in your traditional IRA, and use them to buy assets in the Roth. As you go through the steps to execute the transaction, Vanguard sees that you're requesting a conversion and asks you questions about it, like how much federal and state income tax you want withheld. [ETA: make sure you withhold taxes if you will owe, or the IRS will asses a penalty for underpaid taxes]
Come tax time, Vanguard will send you a 1099-R showing the value of the assets that were rolled over and indicate that they were taxable, as well as the tax they withheld. You report this income on your 1040 with "(conversion)" written on the line. The 1040 instructions explain this. It gets treated as ordinary income. [ETA: you will also need to report the amount of tax withheld on page 2 of the 1040]
Vanguard will also send you a Form 5498 that's for the Roth. It explains how much you contributed to your Roth IRA the previous year and what the source of each contribution was, in this case a conversion from a traditional IRA.
You need to keep this form! Vanguard doesn't provide it electronically and you may need it to make a tax-free distribution from the Roth 5 years later. This is the only proof you have of how the money got into your Roth. Form 5498 is not sent to the IRS like 1099's are, so they may ask you for a copy during a future audit.
[ETA: reference
http://www.obliviousinvestor.com/should-i-keep-my-form-5498/]
I've never taken a Roth distribution so I don't know exactly how that works. I imagine it starts by telling Vanguard to sell something from the Roth IRA account and send me a check, but from there I don't know what will happen.