It's your money. You can leave it where it is if you want when you switch jobs.
Someone did an analysis of HSA accounts a while back. I'll throw a link in if I can find it.
But there are so many variables for how fees affect your returns. I never take money out, so high fees per payment don't impact me, I'm more concerned with investment returns and low investment fees.
As soon as I was able to contribute to an HSA, I did. It's like having a retirement fund on steroids, in addition to the tax benefits l.