Personally, my home value to invested assets ratio is 1:4. To have to replace my home out of pocket would be financially burdensome enough that I'd have to get a job, so I keep my insurance.
Depending on your ratio, or if you're still working and have a high income compared to home value, it might not be too risky to cancel. I doubt many folks fall into this category though.