And this is half the reason I am working towards FIRE. Although, in recent years, i've found some flexibility in my job, working from home 2 days a week, and it helps a lot! Anyway, you aren't alone.
From a perspective side, when I was in college, I thought, "sheesh, working people have it so easy, they just go to work and then can eat dinner and watch TV when they get home." As a college student, I'd go to class, one of three of my part time jobs, be ready to collapse, but need to stay up to study... So it's not so bad. It just gets old.
Then you throw in having a family and life just gets a little crazy, but with the right attitude, it's lovely despite the crazy.