Financially you are fine. Your current assets + pension would cover $135K in annual spend, forever. Not including any money you add between now and then, any income your DW continues to earn, or any pension she brings in.
Yes, it is reasonable to be concerned about college, but (1) you will have a minimum of $25K/yr extra in the budget, even with the mortgage, and (2) it looks like the first kid going away will just about coincide with the house being paid off, which will free up another $20K/yr. Again, even without any extra savings between now and then, that will allow you to cash-flow a lot of college costs and the occasional replacement vehicle.
You are good to go. If you choose to stay, do it because of the people and the job, not because of unfounded financial fears.
And congratulations!