I don't know about your situation specifically, but I would argue that some, not all, but some things do in fact have a lot to do with luck. I feel lucky that I was born into a loving family with educated parents who paid for my undergraduate degree, allowing me no college debt. I feel lucky that I was able to live with them, rent-free, in between undergrad and grad school and also during my graduate internship, thus allowing me to save money.
I feel lucky that I have children that were born with no medical issues that cost hundreds to thousands each month/year. I feel lucky that both my husband and I are in good health and don't have a disease/illness with treatment that costs thousands and we are both able to work (though I am a SAHM, which I also feel thankful about) and we have no pre-existing conditions and can get affordable health and life insurance.
I think some people do have bad luck, through no fault of their own, that has nothing to do with their decisions and that we, as a society, need to stop blaming EVERYTHING on personal decisions. Sometimes life just deals you a crappy hand and you make the most of it...which might lead to not being able to do things like retire early. So yes, those people are going to feel that you are lucky.
That obviously is not the case of many, but unless you are living in their house and paying their bills, then you really don't know.
And, congratulations on your obvious hard work, but also good fortune.