Both Dicey and Swordguy gave measured, thoughtful, responses to your post.
Swordguy went point-by-point to address your issues and then noted that you have "Real fears, and thus they have to be respected and dealt with, but they are still irrational fears at your wealth and spending level."
Errr... what more do you want?
Unlike Bogleheads, here at MMM (especially old-timers) we understand your fears, we've had them ourselves, but we trust the math.
Damn skippy we've had those fears! We went thru some hedonic adaptation AND 2MY (Two More Years) Syndrome before we retired. And it was STILL scary!
The fears we had were real. It took us two years to work thru that no, really, we had God's plenty of money and a damn good plan and I've lost count of how many safety features in our plan -- it was a running joke from readers of my status reports that my hobby seemed to be manufacturing safety features for our retirement plan. Then again, we have a mentally handicapped daughter who simply cannot provide for herself,
so if we got it wrong, she was the one who would suffer, as would our son who would have to support her out of his own retirement savings.
That's why I said your fears are real and have to be respected and dealt with. And it will take time and effort on your part to do that. I would check our net worth weekly or whenever the market moved a fair bit. I got used to watching our net worth change up or down 30-50k in a day. We had already been thru two crashes and never had the slightest inkling of a need to panic and sell at the bottom, so at least we had that going for us coming into the home stretch.
I sat down and said, how will we deal with the market going down and staying down for a longish while?
Answer? Diversify into rental real estate and sharecropped farmland and also have things in diverse stock/bond holdings, i.e., index funds. Let social security be additional gravy rather than an essential part of the plan.
Answer? Have a cash cushion for the first two years.
Answer? Don't let ourselves splurge for at least two years.
Answer? Have lots of optional fluff we could happily live without in our spending plans so we could drop our need to draw down the stash.
Answer? Keep some skills up for a few years so I could get another job if need be. Not really an option for my wife and her career.
Answer? Keep our rentals properly maintained at all times so we would have statistically fewer hits on our stash while we were going thru bad times. Plan for those expenses and attempt to front-load as many of them as possible before retirement.
Answer? Keep our tenants happy by properly maintaining the property at all times and by treating them with respect and kindness.
That's just a fraction of the things we did to harden our plan against the fickle finger of fate.
We didn't just "fear", we used that fear as a spotlight on any possible gaps in our planning. We dealt with our fear.
If the response to fear is just being afraid, it doesn't go away, it tends to get bigger. Drag it out into the light of an adaptable, resilient plan and use that plan to bludgeon it into a mere controlled concern.
At your spend level vs your stash size, you have a massive number of years to adjust and prepare yourself, even if you do have to go back to work. You could afford to get a batchelor's and master's degree in an entirely new field if, for example, your prior occupation of, say, computer punch card puncher, went the way of the dodo. You could even take a year off after college as a gap year. :)
Your SWR assuming your stash is in stocks and bonds is already a very low 2.76% at a $160,000 spend, which is higher than you say you are spending including taxes. That's an extremely safe withdrawal rate. Cutting your spending to $130,000, which including taxes is still probably more than you're planned $100,000 spend, and a full time minimum wage job between you and your spouse, would cut your SWR down to a bit below a 2% SWR from your current stash.
That's why I say your fears are real but irrational.
You've got this finance game won.
You just need to learn to live with it.I hope this longer version of my original message is more useful to you.