If you're putting a footing down, you want to go that far to avoid frost heaves. It's very important for structural building, since you don't want floors or walls to crack or buckle. For a fence, however, I don't think it's that big of a deal, especially if you hammer metal poles directly into the ground or backfill with gravel. They might move a bit in the winter, and if the lateral structure is strong enough, they'll mostly bounce back in the spring. Worst case, you might have to dig up a post every couple of years and reorient it. If that idea bothers you, go down to 4 ft.