If you bought some stock for $100, and it is now worth $300, you have $200 of unrealized gains. Your cost basis is the $100 you bought it for. When you sell for $300, you will be taxed on the capital gains (sale price minus basis) of $200.
Capital gains are taxed at different rates, depending on your tax bracket, and if you have held the investment for at least one year.
Stocks sold after holding for one year or more are taxed at "long term capital gains" rate, which are 0% if you are in the 12% bracket for normal income, 15% if up to about the 35% bracket for normal income.
Stocks sold after holding for less than a year have their capital gains taxed at the normal income tax rate.
Note that realizing capital gains adds to your taxable income, so in the above example, that $200 of capital gains counts as income for tax purposes.
There won't be any fees on Vanguards end, but you will have to talk to your current broker to see if hey have any fees for transferring out, or if the funds you are in have any "back end loads" that you have to pay when you sell.
