It is usually from less self-aware people who treat this as though it is a fluke of the universe, rather than conscious decisions we each made. Like they think the great-life fairy came down and granted me these things, rather than them being direct outcomes of my choice. I'll admit that this aspect of it kind of irritates me. It's like people who make the same amount of money and had generally the same launch into life saying I'm "lucky" because I don't have to worry about money and can essentially afford to buy whatever I want. Luck has nothing to do with it.
I know you meant nothing by this... but as someone who is in their 40s and has no children not by choice, I'd say any assumption in either direction can be, at minimum, annoying. At worst, painful.
Luck has tons to do with it, as it turns out ;) Bad luck, in our case.
(We also thought long and hard about other options like adoption. So it's "lovely" when people present their "ideas" to us, as if we were just unaware of such options, but now they've given us a solution. What a gift!)
I'm sorry for your rough path and the pain associated with that.
I actually did think of people whose family composition isn't what they choose when writing my post, and I tried to make it clear that the people who have said this to me actively choose to have children, and I actively choose not to. That's why the comments annoy, and are simply not true. Because I *chose* not to have children and the choose to have them. I was specifically referring to those who made decisions and were fortunate enough to have those decisions come to fruition, which is why luck had nothing to do with any of it.
In the cases I was referring to and the people who have said this to me, "luck" really did have nothing to do with the disparities they are pointing out. We each made conscious choices and me sleeping in on Saturdays while they go to 4 soccer games is absolutely a direct result of those choices. I know these people enough to know they sought out parenthood and they know me well enough to know I actively avoided it, so no "assumptions" are being made. And luck had nothing to do with the disparities that seem to cause so much envy.
I am certainly aware (and sympathetic) that not everyone's choices around family composition and size are always realized, and that some people who wanted children don't have them and some people have them without having truly wanted them. But that's not what I was referring to.