Yes, it is frustrating!
The only way to control it is to really run the numbers. Vague worries should sound the warning bell, but don't continue to worry without actually sitting down to do the math. You might find that it isn't as bad as you thought, or that you can do a few things to mitigate the risk.
And as other posters said, don't discount the long-term effects. When my mother went back to work after I was born in 1968, all of her salary went to pay for what was essentially a nanny/housekeeper. How depressing is that? But my parents wanted high-quality care (not only did they pay a living wage, but they also paid into social security for my caretakers, which was a rare thing in those days), and they knew they could live modestly on my father's salary.
I don't know how many years she worked "for nothing", but it was a calculated risk. In her field, she had to stay in the game if she wanted a career (and she loved the work). So, years of working "for nothing" did pay off when she began to advance through her career and earned more money. She ended up earning far more than my father when she retired. (And they still live off my father's salary!)