I think that you have more control over what you enjoy than you think.
Some amount of enjoyment in work is going to be related to whether or not you are good at it. Often, if you suck at something - the job sucks.
Also, the work environment - hours, coworkers, heat, cold, etc. will have a huge impact on your enjoyment.
I was put into a new position a little over six months ago that I did not want. I was pretty miserable for 3-4 months because I did not know what I was doing. I've got a better handle on it now. I still don't love it - yes, I have skills in this area but it's not really that FUN. And I'm overworked. But for the most part - I have decent coworkers, and a flexible schedule. I've decided to make the best of it because right now, it's easier than looking for a new job that would expect many more hours.
As far as quitting high paying jobs after sometimes - negating being passionate about something if you get good at it - people quit for many many reasons. I'd assume people who get good at something and quit are likely quitting because they don't like office politics, they are promoted to a "less fun" job, they aren't a fan of the work hours - and honestly, if you are paid well and a good saver, then you have options.