The short version is that you are fine and can give notice today if you want to. With a 4-year advance notice requirement, you have plenty of time to figure things out.
If you give notice now and quit at 55, you can either take $70K immediately, or wait 5 years and take $90K. But in both cases, your wife is still working! If you take the immediate $70K, her income will cover the delta in your needs until she retires, at which point her pension together with yours will more than amply cover your needs. If you wait 5 years, then you just need to take $30-40K from your over $1.5M saved to cover the delta between your wife's income and your expenses. I was an English major, and even I can see that that math is not exactly going to be a problem.
At this point, you are working just to provide additional future cushion that you do not actually need. If you both retired right now, your retirement savings alone would cover around $60K/yr in expenses. Throw in $50K in your pension, and you're more than covered -- even moreso once the house is paid off.
The one caveat I'd add: what are your survivor benefits on both pensions? It's extremely likely one of you will go before the other, and you don't want the survivor to see the pension income cut to below what the budget requires. So for ex., if your pension has only a 50% survivor benefit, that might argue for delaying taking the pension a few years, so that your wife gets 50% of a larger number if you die first. OTOH, if it's 100% survivor benefit, then even the $50K you'd get if you could retire today would be more than enough when combined with her pension.
You've done a great job. Now it's time to reap the benefits and figure out what's next.