I always thought that my family struggled with money because their income was just like everyone else. Really they have just been irresponsible....
I wouldn't be so hard on your family. For some people, the plain truth is "they were irresponsible". But for a lot of other people, the answer might be "they were OK with money, but they chose to live in a crazy expensive area for the sake of their job and kids, because they didn't know better."
I know a lot of families where I grew up (New England) who likely made $100k or more per year, but managed to be on the poorer side of the spectrum in the towns they lived in. This wasn't because they were "poor". It was because they were "relatively poor"; they reached for the most expensive neighborhood they could afford, because they thought it would bring them happiness and a better life. Then they found themselves spending quite a large part of their income, just to try and keep up with the cost of living and expectations / spending of people around them.
I don't know that I'd label those people "wildly irresponsible" - most people just don't think about cost of living adjustments / hedonic adaptation when they choose their home. It's such a small thing, but it will probably play a big role in determining your savings rate and how "rich" or "poor" you feel until you leave the area.
What is the line between middle/upper-middle/ and upper class in your mind (based on income.)
The line is around 50k per year - as you said, the median household income. If you make more than that, you should consider yourself fortunate. If you make multiples of that, you should realize you are
extremely fortunate when compared to the rest of the country - no matter what your area's COL is, or what your neighbors make.
Why can our population go through life believing they are middle class when they have an income 4x the median?
Because middle class sounds approachable, modest, and reasonable. I don't think people like to think of themselves as making an unreasonable amount of money, or being unapproachable.
The fact is, if you're making $100k or $200k, you are
not the norm. This doesn't mean your life is necessarily easier or that you're a smarter, better person, but it does mean that you should acknowledge that you make a hell of a lot more money. Most people don't have it so good, and to think of yourself as "middle class" is to grossly understate how good you have it. It also is an insult to the people getting by on $40-$50k for their families.
If you don't like the words "upper class" (which I don't - it sounds like it refers to aristocracy), that's ok. You can just think of yourself as "very fortunate". But please, let's not pretend that households making $150k+ are "middle class", no matter where they live.
Full disclosure: My wife and I make what we consider "Stupid Oprah Holiday Spectaular Money", living in the highest cost region of the U.S. (NYC). We try to be extremely cognizant of our fortune, and - to quote Morning Owl:
...Mustachians who can see this income for what it is -- i.e., a ton of cash, and an opportunity not to be squandered.
...we try to practice gratitude and not squander the money we're fortunate enough to earn.