Karl Marx said that politics is the question of who works for the benefit of whom. In his era, it was a downtrodden serf class working for the benefit of the nobility and clergy. Our era upends this logic though, with people working for what they think is their own benefit.
The folks spending their lives toiling in jobs they hate are doing so because they believe fancy houses and cars and cell phones will make them happy. They continue doing this, year after year, despite all the evidence they are not becoming happy. They consume hours per day of media and social media propaganda reminding them that more stuff will make them happy, and framing life as a competition to have the nicest stuff. The ads and sitcom lifestyles motivate them to go out and hustle more money each day. This propaganda-work motivation cycle is not all that different than communism, except it is more effective in persuading people to work harder than they otherwise would. It is more effective because it tricks people into believing self-benefit is the acquisition of stuff, when in fact a human needs a bit more variety than that to have a good life.
That said, money is a claim on other people's future labor. It is the essence of having other people work for the benefit of you. It may seem wrong to seek control over other people in this way and to become a parasitic non-contributor, but keep in mind that somebody has to do the work required to support a high quality of life. Somebody has to grow the food, drive the truck, run the stores, account for everything, dig the ditches, manufacture the water pipes, make the shoes, pave the roads, treat the sick, etc. Money keeps this cycle of work going, and to a large extent much of the work is done because people want the work outputs of other people (i.e. the stuff). It is neither good nor bad, it is simply the way things evolved to incentivize the work that is necessary to produce our lifestyles. The miserable workaholics are the ones who believe in the system's propaganda a bit too deeply, and who cannot imagine questioning the system's axioms.
What you're contemplating is not to become a workaholic who drives a leased SUV, lives in a McMansion, subscribes to 5 streaming services, and gets the newest iPhone every year while hating their life. You're contemplating hacking the system by consuming a lot less than you produce, and buying investments instead of stuff. Eventually, you hope to purchase your autonomy and freedom by having investments earn enough to support your cost of living (Note the parallels with escaping serfdom. Some things never change.).
But in addition to this opportunity to escape working all your life, there is a threat. In the culture you live in, almost nobody is willing to work for the benefit of others without money. Those without money go without good quality foods or medicines that could save their lives. Those without money are herded into places where it is too dangerous to walk down the street or drink the tap water. Ours is a cruel culture where elderly people are made homeless because they cannot work to pay rent, and millions of children go to school each day hungry. Those who fail to work will suffer.
You will grow old some day, and be unable to work. This culture will treat you quite badly if you haven't amassed some money by that time, to purchase the labor of people in the future when you cannot do the labor yourself. In addition, you have parents, and may have kids. You and everyone you love may face the risk of an early disability or a large financial loss, such as a house fire, lawsuit, or theft. If that happens and you have no money to spend, this culture will literally leave you to die. Ask the people who can't afford insulin. Maybe things ought to be different, but what ought to be won't matter when it's winter someday and the energy bill is due.
Seen in this light, you don't owe this culture anything. Your only reasonable option is to earn and invest money while you can, while rejecting the propaganda about stuff=happiness. That's the only outcome which allows you to have your needs taken care of when you're too old to work or to take care of the needs of others who hit their own pitfalls. Having money gives you the quasi-political power to direct other people's work toward the benefit of people who matter to you, including yourself. It keeps you from becoming another desperate person drawing down the very limited reserves of charity to the detriment of the other charity recipients.
Money and stuff are both tools, and both are often misused. If you have negative attitudes about money, maybe consider whether you have a too positive attitude about stuff. Those people rumbling up and down the street going to jobs they hate in fancy SUVs are sacrificing their actual happiness for the sake of a symbol of happiness sold to them through propaganda. The people in the drive-thru are signing themselves up to suffer the pains of heart disease because they are told it's a great deal. The people spraying chemicals on their manicured lawns will know the suffering of cancer someday. The people who are fashionable today will spend the last 20 years of their lives in ratty clothes, worried about running out of money because they didn't save.
To some extent, it is the stuff we covet in excess of our needs which causes the suffering. If everyone lived like a Mustachian, there would be no need to worry about having enough money, we'd live longer, healthier lives, and we'd be a lot less interested in sacrificing family, friendships, time, and contentment for the sake of "trading up" houses/cars/clothes/jewelry/toys. We can't save the deep believers, but we can extricate ourselves from the trap they're in.