I was thinking when I went to the bank they'd let me pick what to put it in, but they just put it in a regular savings account. I'm not really sure what to do from here- let it grow and then transfer to Vangaurd?
First off, if you have your healthcare through your employer, it's best if your employer can deduct the money from your paycheck pre-tax. If so, you don't have to pay Social Security deductions on the money. But if you don't have that option, then you can open the HSA with whoever you want.
HSAs are very confusing, there's a million options out there. Some of them (like your bank) only allow you to put money into a savings account. Others allow you to invest the money in the stock market, usually through a "brokerage window". And they ALL charge different fees. So if you want to invest your money in Vanguard funds, you are going to have to open an HSA with someone that offers that option (like HSAadministrators.com that you linked). And you are also going to have to pay attention to the fees that they charge.
From my research, the best one I found was Eli Lily Federal Credit Union (
http://www.elfcu.org/solutions/deposits/health-savings-account/). They have no fees, offer 1% on money in the savings account, and also offer a brokerage window that can invest in ETF index funds. To compare, HSA Administrators charges a $45 fee annually, as well as 80 cents for every $1000 you have invested (
https://hsaadministrators.info/hsa-fees). So you'll have to find one that you like (low fees and investment options).