Try reframing these issues. How much better would it be for your kids to have a parent at home with them during these crucial years? Once they're off to college, you're going to see a lot less of them, possibly for the rest of their lives. What an opportunity this could be!
Next, if you're the SAHP, you can use your time to achieve new levels of frugality by picking up some of the household tasks and challenges. For example, you could teach your kids valuable life skills such as how to cook, clean, and do laundry so they'll be well prepared for college and adulthood. If you're not working, you'll have time to research all kinds of money-saving college hacks for and with your kids.
Since you presumably have enough stashed at 4%, just letting it grow untapped will significantly reduce your withdrawal rate, without lifting a finger or adding any new money. Find an easy job that will replace just your take-home pay and do that instead for a few years.
You could also take a sabbatical now, and gear up for a few working years later, when the kids are in college.
Here's a favorite quote: "Retiring too early is a mistake you can recover from. Too late, and there is no recovery."
Don't know who said it, but they were wise indeed.