What you are feeling is totally normal -- and also something to face down and get over for the sake of your own happiness.
I was one of the poorer kids, and I spent any number of years resenting the kids who had a leg up on me. But I have since learned two lessons:
1. Sometimes an advantage is not an advantage. I relied on scholarships/work to get through school. When my brothers went to school a decade later, they got a free ride from the Bank of Dad. Resentment trigger! But the reality is that both of them screwed around and struggled and took extra time (and in one case multiple schools) to graduate. Turns out that the extra pressure I felt through HS and college not to screw it up gave me a drive (not ever being poor again!) that helped me take better advantages of the opportunities I did have. [I actually never saw this myself until I was about 40 and telling this story to someone, who said, "dude, you totally won" -- and I realized, umm, yeah, I guess I did]
2. You can only start where you are. Our desire to have kids was delayed by miscarriages and infertility. When we were debating a second, I was upset, because I realized I would be 40 when he was born, 58 when he graduated HS, 62 college, etc. etc. But then I realized, well, it's not like I can turn back the clock, right? My choice is to have a second kid, now, at 40, or not -- so of those two choices, which do I prefer? Framed that way, the answer was clear (he's now 11).
I also had a period where I avoided going to the gym, because I was embarrassed about how I looked and how out of shape I had gotten, and I was afraid of looking soft and weak -- yes, in front of total strangers. How self-defeating is that? Finally I realized that, yes, it sucked that I could no longer do what I used to do, but the only way to get back there was to start exactly where I was, today, and build it back up.
Sometimes, starting from behind can be an advantage. Other times, you brain can turn it into an excuse to justify failure. The key is to move past the latter and get to the former.