I don't specifically know what drove the oil prices negative. However, I am not sure speculation is involved. :-)..
Negative prices used to happen all the time in the electricity markets when I was last involved in them. Plain vanilla market supply and demand drives that - not speculation. It is incredibly expensive to shut down nuclear plants, very expensive to do so with coal plants, so on and so forth till you get to the gas powered peakers that can be turned up or down quickly.
In this setup, sometimes it is less expensive for a nuclear power plant to pay someone else to take electricity off their hands than to try and shut the plant down. In fact, the plants are required to submit detailed bids to ISO's ahead of time showing the price points at which they plan to operate, with negative prices very much a planned feature of many non-gas-powered plants. No speculation is involved here.
I bet that if you picked up the wholesale prices of all the 8700 electricity pricing nodes on PJM on 4 consecutive Sunday 12AM-3AM hours, you will find a few negative prices in there. i.e. it used to be very common and prevalent when I last looked.
Any commodity that requires storage can have similar mechanics play out. No place to store or very expensive storage + expensive to discontinue operations => sometimes negative prices.
No speculation is required.
Again, my work no longer involves commodity markets. So I don't "know" what is going on beyond just the headlines. But the above is just my educated guess.
Why could excess electricity not simply be "dumped" rather than paying someone to take it away? I understand why you can't simply "dump" crude oil and it must be physically stored somewhere, but why couldn't excess electricity be used for something useful, or failing that, just discharged into the environment?
I don't know for sure, but can speculate.
I think there must be some physical/technological constraints. How would you "dump" the electricity? Some giant resistor load? producing heat? That does not sound very feasible/cheap/easy at the massive quantities we are talking about here. Also think of the capex spending for such a massive energy sink, all the safety issues/concerns - and it does not sound particularly feasible. Utilities are also regulated entities that need to get approval for all capex. They don't exactly operate like a private enterprise. They *can't* look to "maximize profits" - because their profit margin is capped by regulation. So I doubt they would be motivated to do this, or would be allowed by regulators if they were - "social benefit" is maximized by allowing occasional negative prices instead of spending money building unproductive energy sinks.
Besides the physical/business reasons, I think the main problem is that the clearing price is not available ahead of time. The ISO's first mandate is grid stability, they do whatever they can, turn whatever "peakers" they need to, shut down whatever generator they have to, in order to prevent blackouts/blowouts/brownouts. It is only after the fact that they communicate to the market participants at what price did the spot electricity markets cleared. So I don't see how you would plan ahead of time and "know" which hour's electricity will actually clear at a negative price, and plan for engaging the sink.
All utility companies would have a couple of math/physics PhDs running fabulously complicated "stacking models" (each utility has a different name for it) that attempts to predict how the electricity prices will clear. But, as far as I knew (years ago, when I dealt with commodities markets), none of these models are accurate enough to allow for reliable planning off them.
<Cross posted with maizeman, who, as always, posted a simple and accurate response.>