This is a struggle I have been dealing with for the last 4 years (oldest was born in 2015).
I, too, find myself deriving little satisfaction in solo activities. I can't remember the last time I did something (not work related) without my wife+kids around and didn't go longer than an hour thinking about them and the time I'm missing with them.
My own rationalization of these feelings has been a long journey and I'm still not fully comfortable yet, in that after extended periods of focusing on only work+family I do crave solo time (as anyone should) and yet when I indulge in that time I feel as though I should be back at home with them.
My mind's borderline abhorrence of personal time is rooted in two things, which are very similar to yours - a strong feeling of guilt that I'm not fulfilling my role as my wife's (who is a SAHP) backup - something she looks forward to most days, and an even stronger feeling of anxiety over knowing that each second spent without my kids is lost forever.
Being a parent is, has been, and always will be very challenging. And for the primary caregiver (at least, in my own experience), it's demoralizing at times to watch your spouse go off to work every day. I will admit that there are a significant number of things that compound my wife's challenges with being the primary caregiver but ultimately at then end of the day with the boys she's absolutely drained. Extreme temperatures, rain, snow, illness, judgmental other parents, mean other kids, bad moods, and a whole host of other things can make the day of a SAHP so much more challenging than most jobs. And so, I am extremely glad to take the burden off her shoulders. And when I go off to do something on my own, especially without the kids.....I feel as though I'm failing in my responsibilities as the other caregiver. Just as she feels the same way whenever she goes somewhere and leaves the kids with me.
And when I'm not feeling like I should be home to relieve her, I feel like I should be home with my kids. To watch them play, learn, grow. I've watched my 4yo, who was only recently a chubby little infant, turn into a little boy with boundless energy and humor. And I feel anxious because these moments are fleeting, especially since he's going to be in school soon and in 9 years he's going to want little to do with me and my wife. Time moves faster and it's impossible to not feel like I'm on the losing end of creating memories.
We're dealing with our challenges in our own way, and some people just don't feel the same way. Some people have no issue leaving, for a few hours, for a weekend, or (in the worst cases) forever. And others don't care about spending those early childhood years with their kids. So, it's difficult for a lot of people to relate to what you're feeling.
There's no one-size-fits-all answer to this internal struggle. I've allowed my hobbies and interests to take a backseat for the time being, at least as long as my kids are too young to enjoy them with me. I've learned to better embrace my role as a father and incorporate that into my identity rather than it be an annotation in my self description. Doing this has eased the guilt a bit and allowed me to discover that some of the things I thought I enjoyed doing really aren't enjoyable anymore.
This is all going to shift again (as it probably will for you) when all my kids are in school. My wife will "have her days back" and I'll not feel so bad when I decide to take a Saturday morning for myself. But ultimately, what keeps me centered is the thought that, when my kids are graduating high school, I don't want to watch them and regret never learning to enjoy my time with them. If that means fewer beers brewed, fewer photos taken, fewer games played then so be it.