Honestly, it's uncertainty about the future.
The formative years of my professional life were 2008-2010, at the height of the Great Recession. I am crystal clear on the fact that jobs can evaporate out from under anyone, at any time, for reasons well outside their control. My husband works in a high tech field and worries that it will become so complex over the coming decades that he'll have a hard time keeping up. I'm in a great situation right now, but all it would take would be a change of leadership in my company to someone who says, "Meh. We don't need that group. The other functions can absorb that work."
As long as you're reliant on your wages to live, you have to accept that you're in a precarious position that could change at any time. No one is indispensable. There is no such thing as loyalty in a company. If someone decides you're not worth the line you take up on a spreadsheet, you could be out tomorrow.
And in the long term, I worry about the impact of climate change on just about everything.
Right now, we're in a position to 1) build a giant pile of money that can support us for the rest of our lives, 2) buy a second property outside our coastal, HCOL city where we can further indulge our gardening habit, and 3) learn skills and habits that make frugality automatic and joyful. In a few years, we'll have enough that it won't matter whether we ever work again.
That security--that we're good no matter what--is very motivating.