The friend adviser wasn't actually that good of a friend if he never touched base with you and got that account growing. I'd imagine that the funds were in some nice high expense ratios as well, and that's definitely not a "friendly" move either. That's very poor service to you, but great for him and his company. I wouldn't actually talk directly to him about it - just contact Vanguard and have them initiate the roll. If he calls you or anything, just tell him that it's nothing personal, just that you're trying to take a more active roll in your retirement and his company wasn't a good fit for you, and that you prefer not to discuss it any further (if you want to keep the friendship).
As far as moving an account, it is really, really easy. Filling out some forms, get it notarized (or sig guarantee) and send them off, and they take care of the rest. Sometimes it can be done online if you're lucky. Vanguard is a great company, and you'll probably be very happy with your accounts there. From what I understand, they are a more individual investor DIY vs. customer service orientated company (nothing wrong with that).
Sounds like you are doing pretty well tho, so congrats. :)
We'd probably need a bunch more info on the company retirement program to make any informed guesses. I'd be concerned that it sounds a bit like a pension maybe? In which case I'd be more concerned because smaller companies (especially family run) can be more likely to go out of business and take your money with them. It would require more research on your part to figure out exactly what the program entails, and what it means if you leave the company or if they get bought out or go under.