My boys are a lot younger than yours (5,3,1) so YMMV, but we've never been able to roll over more than a few hundred dollars year to year.
We have an HSA in the first place because for our circumstance, the HSA nets out to the cheapest plan. We have 3 plan choices at work and the HSA plan is the 'middle' choice in terms of premiums that we're responsible for but it's highest in OOP costs and deductibles (understandably). This is made up on the back end - my employer contributes $2k to our plan annually, so with that the net OOP cost is slightly lower than the other two plans.
We contribute a few thousand pretax to the plan as well, but births, ear infections, croup, medications, dental costs, vision costs, etc. all whittle down the HSA balance. We're absolutely not in a position to be contributing to the HSA AND paying OOP for everything, so for us it's not a retirement vehicle at all right now.