Money isn't the problem, you have enough. The question you're asking is, "Do I cruise through the next five years on autopilot, or trade in my safe, easy gig for something harder but potentially more rewarding?" That's really a question you have to answer for yourself, but there are ways to evaluate the risk.
Weigh the odds that the other gig will be better. Is it a bit of the "grass is always greener" problem, or is there a real potential for higher satisfaction? Draw up some scenarios - make a worst case scenario and a best case scenario, then develop a "most likely scenario" that will be somewhere in the middle of these. Use the worst case scenario to evaluate the risk. Quantifying the scenarios helps us make decisions as we think through the details that we don't know, but can predict.
On the other side, there's nothing wrong with sticking to an easy path and coasting into your early retirement, either. You're working 43 weeks a year at a job that requires five hours a day, four days a week. Americans in private industry work an average of 2300 hours a year. You're putting in just 860 a year. You have a part time job for full time pay, plus a guaranteed pension and platinum plated benefits. Color me jealous.
In both scenarios you have enough of a stash to generate 41k a year in inflation adjusted income for your retirement, from your 1.04m in assets. Your pension will generate 30k a year if you were to leave in five years. Assume you'll also have social security available at 65 (or whenever you feel like taking it). Your expenses are likely to go down once your kids move on. So you need, at most, 60k a year but most likely less, and you'll have 71k a year in five years. You're in great shape.