I assume you do not qualify to deduct the $5,500 contribution because you are over the income limit. If this is the case, there are three reasonable options:
1. Call Vanguard and tell them that you want to withdraw your excess contribution to the traditional IRA plus any earnings (AKA net income attributable). You will have to pay ordinary income taxes on any earnings plus a 10% penalty on the earnings if you are under 59 1/2. You will not be entitled to any deduction or credit for the IRA contribution. Note: This option may have gone away once you did the rollover.
2. Leave it in your Rollover IRA as a nondeductible IRA contribution. You can take any deduction or credit on the IRA contribution on your taxes in the year of contribution. But from now on, any conversion to your Roth will be treated as being taken pro-rata from your deductible and non-deductible portions of your rollover IRA. So if you already had, for example, $55,000 in your rollover IRA, then for every $11 you move from your traditional IRA to your Roth IRA, $1 would be from the $5500 in non-deductible (and would not trigger any taxes) and $10 would be from the $55,000 in deductible (and would trigger taxes, as Vanguard hinted).
3. Undo the contribution as in step 1, then roll the Rollover IRA into your current 401(k) if it is allowed. Then you will be able to do a backdoor Roth next year. There are timing considerations in doing this, and you may not have a 401(k) or your 401(k) may not allow rollovers.