Based on the steep topography of the land and difficult weather conditions at 11,000 ft., I seriously doubt it would be profitable to extract gravel at that location in my lifetime.
My worry wouldn't be that someone would open a commercial gravel pit on your property, it would be that the seller stays in touch with an old buddy up the road who happens to own an excavator and wants a few yards of gravel every year to resurface a driveway or access road. Seller tells his buddy he's welcome to go in and dig what he wants for some nominal fee. Now you've got an amateur occasionally showing up to drive across your property without notice and dig up random spots.
The potential for a low level annoyance seems real even if there's nothing on the property that's commercially viable. There's a certain sort of dude who will make it his mission to get something out of these rights just because he can.
It wouldn't allow a gravel pit, or someone digging up sand or rock or gravel; that would be considered surface use.
This would be for something that is truly underground, like oil or natural gas.
The problem is that the owner of the mineral rights could set up equipment on the surface to test for and extract the underground resources, without your approval.
Around my neck of the woods there's no drilling of gas or oil (a couple miles thickness of hard basalt from ancient lava flows tends to discourage drilling).
However, there are underground caverns in the basalt that are used for natural gas storage in the nearby hills. The gas company buys low, fill the caverns, and sell high, similar to the strategic petroleum reserve.
One rumor I heard years ago: the gyppo logger that previously owned the hundreds or thousands of acres was in the hospital with dementia and an unscrupulous representative of the gas company got him to sign away the mineral rights to the land, which was worth much more than the trees on the land due to it's potential as gas storage, which they then developed.
His family swore he never would have signed off if he'd been in his "right mind", they tried to sue and lost.
So now, when old timers around here sell off their hundreds of acres (after it's been logged, of course) many stipulate that they will retain the mineral rights, because "won't get fooled again!".