Definitions of class are usually based on:
1) Income stream
2) Income source
3) Income perpetuity
4) Income spent
There are pros and cons to every one of these definitions that would cause a person under one definition of middle class to move to upper class or lower class under other definitions.
Under (1), a person has to make a certain level of income to qualify. But this ignores the fact that a person might face extreme hardship in the event of a job loss or personal tragedy. In the words of Millionaire Next Door, this person is Income Statement wealthy and could face difficulty if a continued stream of income is the only source of financial stability.
Under (2), a person's income is mainly derived from the investment of capital rather than the trade of hours for currency. But this implies a higher class for a person who is willing to live off $20K annually over the person who is still earning $100K annually. Per Millionaire Next Door, this person is Balance Sheet wealthy and is probably more secure than the strict wage earner. Of course, security here is also measured by how close a margin this person cuts his budget with respect to his revenue stream.
Under (3), a person's income is derived from the investment of capital, the hourly wage/salary, or both. Once a person is truly FI, then perpetuity (or a reasonable estimate of perpetuity) is assumed. Again, this might then become a (2) situation, but then the same $20K vs $100K issue recurs.
Under (4), a person's status is defined by his/her possessions and experiences. Since these are somewhat correlated to income, a higher earner in (1) might also be a bigger spender here. This is the criteria the FI/RE folks living off lower income via moderate investments would often fail. They choose not to chase extravagant things and/or experiences, but many people would say those folks are just unmotivated--similar to a "trust fund baby." When a person is significantly younger than typical retirement age and voluntarily lives off an income less than the American median, much lower than their peers, and/or much lower than they would earn if still working, then many would classify that person into a lower class simply because of the apparent lack of spending and the absence of a typical middle-class lifestyle.
Personally, I tend to think that a person who is FI or on track to become FI and can manage daily finances with a reasonable amount of "No" is middle class. If a person is FI and doesn't even have to consider "No" for most decisions, that person is upper class. For me, this bases status on net worth, passive income, and above average ability to spend--all at the same time. Whether a person chooses to work or chooses to spend at that point is really irrelevant because they really do have a choice that is not predetermined by their financial situation.