Required off-street parking is a subsidy to drivers, at the expense of non-drivers. The whole point of these policies is to keep the on-street parking free and plentiful. To achieve this policy goal, denser development cannot add
any net new cars to the on-street parking situation. Building the parking is a necessary but insufficient condition for meeting this goal. The residents also need to choose to actually use it. For this to happen when there's free street parking nearby, the off-street parking needs to also be given away to residents for free, or nearly free.
The problem with this is that off-street parking is expensive to build. If the residents of each apartment in your building all own exactly the same number of cars per apartment, this isn't a big deal. The cost gets baked into the rent and everyone ends up paying their fair share. If some residents own more cars than others, the cost is still baked into the apartment rent, but this time the people with fewer cars are paying for garage spaces their neighbors are using.
This is exactly backwards! If we want to reduce traffic and carbon dioxide emissions we need to give people financial incentives to reduce their car ownership. Instead we have a system where the express goal is to set the market price of car storage as close to $0 as possible, spreading the infrastructure costs out equally among everyone whether they own a car or not.
What you are talking about is exactly about developing commonsense zoning, building as well as parking laws.
My idea of a commonsense parking law would be pretty similar to what Japan does: if you want to own a car, you need to prove that you have a place (off the public streets) to store it at night. Street parking, where it even exists, is for short-term use only. Cars parked on the street overnight are towed.
What could be more commonsense than that? For any other type of property I might want to own, it's up to me to find a place to store it. I wouldn't expect to be able to store it on public land, and I certainly wouldn't expect storage to be given away for free by the city. Why should cars be an exception? Why should our zoning codes be in the business of forecasting exactly how much parking the occupants of a building might need? Why not let the building owners figure this out for themselves?
I think this system puts the incentives in a much better place. Want to live with a car? Fine! Find a house with a garage. Nobody is stopping home builders from including as many garage spaces as they like. Want to live without a car? Great! You'll save money by getting a home without a garage you don't need.
Now, I don't think it's practical or desirable to implement full-on overnight parking bans on our streets in the US. Our existing streets are designed with some parking in mind, and many existing homes are occupied by people who have come to rely on that. However I do think that going forward we need to recognize that storing your car on the public streets for free is not a human right.
Charging a market-clearing price for overnight parking permits is what I would suggest for any street where the on-street parking fills up. This would help keep the space available for those who have fewer alternatives, as it would create an incentive for people to use garages and driveways they already own instead of the possibly more convenient street parking. It would also create incentives for people to create more off-street parking on lots where there's room for it to be done cheaply. Do this and we'll gradually transition to a system where the amount you pay for parking (whether on-street or off-street) is pretty close to the amount it costs to build and maintain a parking space. Seems fair to me.