I view my financial health like my physical health. There are times when I'm super disciplined and times when I'm more relaxed. Times when I'm motivated to work more (on money or health) and times when it feels better for me to simply focus on cutting out excess (Again, on money or health). Both are progress. This is a lifelong journey, not a prison sentence to be endured for a time and then relieved from.
It seems like you're delaying gratification so much that you're unhappy with your current life, and that's going to have a bad outcome. Or perhaps you feel an imbalance in responsibilities where you're responsible for the bulk of the finances, and your wife is responsible for the bulk of the childcare. In that case, perhaps a more even sharing of those responsibilities would bring more contentment. Either way, I'd suggest thinking about what you plan to do in FIRE that you think will allow you to "really enjoy life", and then consider ways to make that happen in your current life in smaller amounts. The path toward FIRE isn't about years of exhausting self deprivation to later enjoy excess. It's about reducing or eliminating wasted time/energy/money that isn't making you happy or propelling you forward. Life is a balance, and clearly you feel unbalanced at the moment. So do a bit less of the thing that's making you unhappy, and a bit more of the thing that makes you happy.
I'd also suggest that with a time frame of a couple of decades, a lot can change. It's terrific to have a plan, but if your entire focus is something so far into the future, you're going to miss what's happening in the short term. Your wife being a SAHM while the kids are young could make tons of sense and be super beneficial for your entire family at this stage. But once the kids go to school, she may find more time to generate additional income. You may stumble upon an opportunity at work or for other investment. You may get some other windfall. Any of those could shorten your time frame.