I'll answer these as candidly as I can. I realize that I certainly have benefitted from systemic items in some areas.
Would your perception have been different if you were a woman and felt like you were being harrassed by customers, coworkers, or management?
I have been sexually harassed in a job before. I was a server in Austin, TX, in an area with a large gay population. I was flirted with frequently. I never dissuaded the behavior, because I knew that I would get bigger tips to go along with it. I realize that women face this far more than I ever did, and I never worried about someone waiting for me after work. That's the only job where I ever dealt with that.
What about being a minority (apologies if you are)? Don't you think that the expectations would have been different, especially at the lower wage jobs?
In the part of Texas I grew up in, the Hispanic population is actually larger than the white population. Of course, that doesn't mean the system favors them over white people. Most of my bosses were white. In truth, the closest I've ever felt to being a minority was being an outspoken atheist in the Bible Belt. However, I don't think my Hispanic co-workers had a harder time at work than I did. The black population was under 5% of the population, and I actually had more black supervisors/managers than co-workers in my low wage jobs. Sample size is too small to take any meaning from that.
Edit: This is a stereotype on my part, but I worked with many first generation Americans, and I often found them to be harder working than many of my white co-workers.
How about having kids to look after, or aging parents, or both? Your inflexibility to pick up shifts might have been a nuisance to your managers.
I acknowledge that. I would have been less willing to work more hours if I had a child back then. However, all of my managers have been parents, as far as I can remember. I'm sure they understood that kids come with a time commitment. I don't even recall any of my managers ever expecting people to work hours they weren't scheduled for, because there were always employees like me who would volunteer for the extra hours.
That does remind me of my time in the military, however. They actually care about family time so much that the single soldiers were often stuck with the shit details. And it wasn't like we got paid more for doing them.
I did have an assistant manager who took care of her aging mother. Being in a management position, she did have to work extra shifts (have to have a manager on duty), but the manager never asked her to do work that he wasn't also willing to do, and he had a very sick wife as well.
What if you had a disability, visible or hidden? Do you think you would have been treated the same way?
I'm trying to picture what you have in mind here. I don't think that I've ever had co-workers with disabilities, but I have a difficult time imagining my past employers treating them any differently.
I'll emphasize, I believe there absolutely should be an expectation that when working, one should strive to meet at least the minimum expectations of their position, to go above and beyond where appropriate, and to take pride in the work they do. But hard work isn't enough, luck is a major factor in being able to enjoy a job - whether you make $8, $15, $50, or $150 an hour. And when folks who are doing their best are treated like crap whether that's at a fast food establishment, a fulfillment center, or a software engineering job, any opportunity to improve upon your current position is going to be appealing.
I don't think that it's a product of my skin color, gender, parental status, or lack of disability that has led to managers treating me well. I think I've just had good managers for the most part who treated everyone well (excluding the military).