Obstructionism is only part of the problem. Part of this is our actual ability to build and fund. It is not political and is bipartisan, though it has roots in politics.
Our public works and infrastructure process and institutions don't have institutional knowledge or ability, and turn to the private sector. It's sort of like the healthcare system where we solved the problem by letting corporations benefit. Everyone can get coverage, the insurers are happy, the doctors and hospitals and national debt are not.
No agency has in-house experts, as both public agencies and private companies once did. Both once had staffs who could plan effectively and efficiently in-house, draw up contract docs, and let work to contractors who would bid to build it, or sometimes to design and build it, in the conventional bid-build practice.
Now we turn to consultants for everything, and the consultants in turn have very little location specific expertise. This happens at planning stages for operation and construction, and again during construction. All design and construction work is huge design-build joint ventures that guarantee profits for the consultants even when nothing is built. No one is ever at risk.
They also have little stable funding. All these agencies had their funding cut over the years to reduce pension obligations and because "government can't do anything right". What ensued was a huge handout to private design and construction firms in the AEC industry.
It is really hard to plan and build when you don't know if the funding will materialize. Some agencies still issue bonds for specific projects but when it gets politicized it's a mess. In addition, as funds dwindle, agencies focus what elected officials want in return for the support the agency does get. For transit agencies they also focus on unprofitable routes and service to function like a welfare agency. As you would expect, when a service provides the bare minimum, it gets abandoned by anyone with choice, reducing revenues further.
And some public works projects don't begin with actual need or cost-benefit analysis. Often there's a big push to do a project because a mayor or governor or county official selects it. If a transit agency has three or four needs, they seldom do the invisible thing that provides an incremental upgrade if there is instead a big expensive legacy project an elected official can hype. Same with highways and bridges.
There is also cost inflation due to known future lack of maintenance or known future inability to fund/ build more. Projects get overbuilt because everyone knows there will be no money to care for the thing once built. They get overdesigned because everyone knows that if you don't put in the capacity now, which you may never need, you will wait 30 years for any funding to address it. Instead of good engineering, fit for purpose and efficient, you get projects that are massive, and you get few of them. No reason to change: the agencies cannot and just want to keep the tap on even if only dripping; the AEC industry, dominated by the big design-build firms, are well compensated; and political leaders don't know enough to challenge any of it. It's sort of bipartisan patronage politics I guess.
I know someone who worked for one of these design consortiums once on a late project phase. The budget kept shrinking due to cost overruns on earlier phases (designed by the engineering consortium) and the cost kept increasing. The solution was to bring in an oversight consultant to the engineers, and the engineers' design fee was cut to pay for the oversight. No additional funding was in the pipeline for design or construction, and a new presidential administration redirected transit funds for the upcoming year, so in the end nothing in the later phases was built due to the earlier overruns. The contractor was made whole. The consultants were OK, especially the oversight folks, who ended up paid to oversee nothing. The agency got only the first phases and an incomplete project. All declared victory and ridership never reached expected levels because the thing didn't do what it was designed to do and didn't go where it was designed to go.
The whole thing is bipartisan at this point, but has its roots in the increased role of government in building infrastructure that came with the auto era and demise of railroads, followed by distrust in government and its expertise during Reagan-Bush-Clinton. This will last as long as we believe government = bad/ private = good, as long as we fund this public-private nonsense that privatizes gains and socializes losses, and as long as our politicians are happier to strangle government ability and expertise at the expense of getting things done.
For actual detailed analysis check out the Pedestrian Observations blog by Alon Levy, subtitled "For Walkability and Good Transit, and Against Boondoggles and Pollution".
https://pedestrianobservations.com/author/abstractnonsense/Unfortunately other countries seem to be adopting our system, under lobbying by multinational construction firms and US-led development expertise.
