The system reinforced itself.
Yeah, it's almost as if when you socialize the costs of low density (highway construction, pollution) and privatize the benefits (more space, exclusive schools), lots of people will take the hint.
If the carrot also needs a stick, we'll concentrate social dysfunction (housing instability, addiction, and the resultant crime) in walkable locations and localize the costs of dealing with it.
Between the carrots and sticks, I get how lots of people eventually saddle up for the exurban commute.
Glad you mention social dysfunction as a cause for people spreading out into low density cities. Just ask a resident of a suburban sprawl area why they don't live in the city core, and they'll explain how the crime, beggars, addicts, etc. make it an unpleasant place for them and their kids. Gated suburban/exurban communities are telling us that people feel insecure.
It seems lots of people have a vision for what a functional high density city would look like, but this vision cannot be sold because the problem is other people's behavior. I.e. if you build it, the addicts will still be there hassling people on every street corner, the gangs that sell them drugs will still be committing acts of violence, etc. The drug trade, being underground, is more efficient in dense areas.
Our inability to address the problem of substance abuse and addiction is the root cause of sprawl, car culture, and architectural dysfunction. HCOL and VHCOL places with high urban density "work" to the extent that economic segregation causes a steady outflow of addicts to less expensive places. I.e. if you are a heroin addict, paying $3k/mo in rent and holding down a job that can cover that amount is that much harder. You'll eventually end up in a cheaper place, no less addicted. This cheaper place might be a "ghetto" in a less-expensive city, which only discourages people from investing in such cities. Thus, the urban life in SanFrancisco or Boston is prohibitively expensive, and in places like Tulsa, OK, Memphis, TN, Montgomery, AL, Kansas City, MO, Cincinnati, OH etc. you can buy a livable house for under $100k. Guess where the addicts are going, and guess which cities are becoming hollowed out and sprawling.
If addiction treatment was offered to everyone on the taxpayer's dime, addicts and the drug trade would become less common. Taxpayers then might be spared a lot of emergency medial bills, road construction, commute costs, gated community HOAs, law enforcement, utility infrastructure costs, etc. and in addition there would be more taxpayers to carry this burden. Unfortunately, the American way of thinking would see this as somehow rewarding their behavior. We'd rather pay the enormous expenses we're paying and live with dystopian hour long commutes than having a nationwide rehab program. If we only got half the addicts off the streets, LCOL cities would become plausible places again.