On my road trip across the country this summer there were several places that the high clearance of our van was required. The worst road was the access to the trailhead for Wetterhorn and Matterhorn in the San Juans, and there were a few others.
In Washington, the access road to Mt. Adams is quite poorly maintained, and high clearance comes in handy there too. I have seen people with Subaru Outbacks and such up there, but a Prius' ground clearance is way too low to make it.
The only free camping area I know of around Leavenworth, Washington is a couple miles up a narrow dirt road full of potholes. I think I went up there twice this fall, and would have gone to that campground more if I had known about it sooner. I have similar stories about looking for camping around Boulder Canyon in Colorado.
Several ski areas in the northwest, especially backcountry, are inaccessible to FWD sedans when there's lots of snow. Also, the snow chains available for tiny sedans are a joke because there's not enough room in the wheel well for anything except tiny cables, and you need hands the size of a Japanese person to put them on (speaking from personal experience).
I'm with Jamesqf that some serious outdoors people cannot get by with unmodified subcompact cars, and if the vehicles we need are used for nothing other than frequent access to the backcountry, it's not an inefficient usage.
I'm not advocating for SUVs though, those things are just dumb. My dream vehicle is a Suzuki SX4 with a lift kit, that would be perfect for all of the above and still get acceptable mileage. A Geo Tracker should also have no problem with any of the areas I've mentioned and it would be much cheaper.
