The drive is tough, but personally I love a side hustle and has helped us get far ahead. Ever since I graduated college I've had a side job of coaching and all of that money has either gone straight to investing or saving for our condo and then house. No way I would have been able to save that kind of money and invested that much if it weren't for my side job, as I never spent any of it. Can you do this side job closer to your house or get the same side job somewhere else? If you are basically wasting the day on a drive for 5 hours/week (I assume it's all in 1 straight shot) then can you make the most bang for your buck and work 8-12 hours in that shift? Since the day is basically wasted you might as well get the max benefit. I used to take private training sessions and stuff, 1 here and 1 there, and I finally standardized it and told people "this is the block I'm doing it in" knowing that those private sessions could go away, but they were on board, made my life easy in that regard, and I can knock out a bunch back to back, so that has helped me be more efficient and keep the easy customers.
I love the side job but not the driving, so maybe find a way to cut that down or get the max benefit from it, like maybe you can do half of it from home and you do admin work 1 week and then on-site work the next week? Not sure what your side job is but maybe it sounds like something in the construction business?