Are we talking completely alone, cabin in the woods like? Four acres is a lot of land, but not terribly big if you have neighbors around. If that's the only house for miles around, though, I understand your reservation.
Not too far away from me, there are two prisons - medium security and maximum security - that also have schools near by. Not next door neighbors, but enough to make you go, hmm.... But maybe that's almost better, since there are plenty of people (and police) around.
I wouldn't be too keen on living in a house with zero people around for miles regardless of the nearest prison, but that's just me. I think as long as you have a decent security system that you're comfortable with (whether it's ADT, a loud scary sounding dog, a shotgun, whatever), I think you'd be fine.