Until recently, at my last job, it was any one of:
- Having periods of time with no work, in which I still had to justify my use of time
- Then being given a task, and an unrealistic deadline, and having someone metaphorically (and sometimes literally) breathe over my shoulder until it was done, and feeling like I had to justify my use of time
- Being ordered around by one of the worst people I have ever met
- Having him think he was my boss, even though he wasn't
- Having other people think he was my boss, even though he wasn't
But I quit and got a much more pleasant, less stressful job. As far as I can tell, I like this job, as far as jobs go, so the normal worst things about working for me now are:
- Never being sure I am doing well enough (imposter syndrome ... or maybe I'm just really terrible)
- I'm quite a low-energy person/need a lot of downtime, so even with a sensible work-life balance (37.5 hours a week, some degree of flexible time, commute of 15-30 mins each way, 25 days holiday a year) I feel like work + gym + housework + errands takes up all my 'productive' energy and I spend the rest of my time just lazing around to recharge. Don't know if this is an accurate assessment or if I would be just as lazy if I didn't work!