There's no denying systemic problems that may hold people back, but I find it hard to believe that ALL success is based on "luck" or "privilege." After all, there are people who have strikingly similar demographic information as the OP, who had as much or more help, who are not millionaires. We see people make poor decisions every day. It's not bad luck that made them make bad decisions, it's something else.
Of course, the mystery of why some people are successful while other people from similar circumstances are not successful is a question that social scientists everywhere are trying to solve every single day. [I recall that my mother's PhD in Sociology was on the topic of why students in the same high school were choosing to make certain choices about what classes to take. Why did some students put themselves on a University-oriented track, while others made choices that eliminated many of their future options? This was a time when the Ontario school system had a 13th, university-prep grade called OAC and OAC classes were required for entry into University, but were optional for students to graduate. I don't think she solved that mystery, but my recollection was that assumptions that the student made about their own intelligence as early as elementary school were framing their class choice in high school, and those assumptions were heavily influenced by the words/opinions of teachers and parents, rather than grades.]
I am not a social scientist, so I don't want to speculate too deeply into what makes some people successful, but one thing I have noticed is that not everyone asks themselves "How can I improve...?" The end of that question is different for everyone, "How can I improve efficiency?" "How can I improve my skills?" "How can I improve my career?" "How can I improve my budget?" but even asking the question is a starting point. Eliminate from "success" all of the people who will never ask the question, because ambition and goal-setting is not a fundamental part of their personality.
Then you have all of the people who make the wrong choices. When they ask "How can I improve...?" it leads them to MLMs, get-rich quick schemes, poor investments, spending too much on an elite education, bad business decisions, etc. Eliminate from "success" all of the people who simply choose the wrong path to success because of... reasons. They tried. They failed. It wasn't because of luck, because many bad decisions are avoidable via self-educating, and it wasn't lack of privilege, but it also wasn't because of lack of ambition or lack of ability.
Eliminate from "success" all of the people who truly can't. Mental illness, physical disability, mental disability - some hurdles can't be overcome with grit and effort. I have a married couple who are my clients who are really nice people, but of below-average intelligence. They are on that fine line between mentally competent and not mentally competent. As a result, they are frequent victims of scams and fraud. A work ethic just isn't enough. Trying hard isn't enough. Because someone is always around the corner waiting to take advantage of them. And when they have a problem, they are not capable of navigating a complex system to solve their problem and they don't have the social connections to seek help. For example, when someone stole of the money out of their bank account, causing it to repeatedly overdraft with a huge amount of overdraft fees, the bank basically refused to help them because they were unable to communicate the problem accurately and were dismissed. Do you think I would be dismissed if I was in a similar situation? Absolutely not. Indeed, I was able to contact someone I knew at their bank on their behalf and ask for assistance. Not only do these people have more problems than normal, they lack the ability to find a decent solution to their problems, which makes the consequences much, much worse than they would be for you or I.
My husband was poor. He was also subjected to religious abuse. His mother took him out of school and homeschooled with a bible-based education. He stole books from the library because he couldn't get a library card without his parent's permission and she refused, because the bible was the only book he needed. When she found the books, she hosted book burnings. She made sure anything else he liked was thrown in as punishment - stuffed toys, comic books, cassettes, etc. He stole food from the grocery store because he didn't get enough food to eat at home. He moved out at 16 and had bad grades in school because he would fall asleep in class because he had to work night-shift to support himself. What made him become a non-college educated, successful start-up owner, when one brother became drug addict and the other became a college-educated teacher? I have a hard time calling him "lucky." But if you ask him, it was luck. He thinks that since he was able to remember a better time, before the extreme poverty and religious abuse, he understood that it was wrong and therefore actively worked to get out. His middle brother did not have solid memories of a better time, and therefore couldn't figure out what to work toward. His youngest brother experienced the least abuse and poverty, and his upbringing was a bit more conventional, so he turned out more conventional. He considers himself and his brothers basically three versions of the same person - equivalent intelligence, same genes, same town, same educational opportunities, same family, similar personalities - but born a year or two apart, they lived slightly different experiences and therefore made different choices that led to various levels of success or failure. He thinks having a good business idea at the right time was lucky, too. And yet in Grade 1, prior to being yanked our of school, when he was asked to write down and draw a picture of what he wanted to be when he grew up, he said he wanted to be "retired." His goal throughout his youth was to be rich and retired. So it can't all be luck, right? Something in his personality caused him to want that.
I don't think it's right to diminish the hard work of successful people. Successful people have often made sacrifices or taken risks that non-successful were not willing to entertain. Yes, it is likely that privilege and luck played a role in that success, but it does not explain 100% of success. So should you be grateful for the role luck and privilege played? Absolutely. And you should acknowledge it and respect the role that lack of privilege and lack of luck has played in the lives of others. But you also deserve to be proud of your success. It is something that was born of your effort and your choices. Other people may not have had the same choices as you, they may have lacked choice, or they may have been forced to make harder choices, but you did have choice and you made the most of it.