Many of the very real neighbor problems listed above can be mitigated by buying more land than the five acres you mention. We live on 25 acres a little over an hour and a half from any "real" city. We do, or did, have the alcoholic methhead neighbors, though if there was any molesting going on, I was blissfully unaware. They all seem to be in prison now, which is happy-making. While they we the, we only saw are heard them if we drove out, and then only if we went in their direction, and they were "next door." I did have to fend off what I believe to be one of their customers one morning shortly before they disappeared, so it can happen, but that was once in five years, trespassing at great personal effort (it's a hell of a hike up here).
Trespassing and burning are harder to mitigate, though at distance the burning is only an issue if they let it get out of hand. We had one memorable case of this - we got some unplanned bulldozer tracks on our property and they got a big bill they surely didn't pay from the forestry service.
Best advice? Take your time. Start reading books. If Rob Roy wrote it, it's good. If Rodale published it, it's almost certainly good. Check out the Mother Earth News website - some of the older articles are better than recent ones (which are good), and the website goes back to at least the 70s.
Take time with the land, too. You don't have to get excited about any particular parcel for sale, not with your timeline. So don't; just start evaluating whether parcels fit your needs. Even if you're not ready to buy yet, start following the listings (or, better, look around for "for sale" signs, too, if you know the area you want - rural land is often sold without a listing).
Find out what the health departments where you want to buy require for a septic system (perk test is the old version, soil map is more likely). It'll bother you to do it, but pay for that to be done before you buy, especially if you only buy five or so acres. If you have to start over because a place won't take a septic system, you'll lose a few hundred dollars. If you buy a place and then find you can't build, you lose your housing fund.
My husband and I started our search with a list of things our property had to have, things that would be nice to have, and things that were deal-breakers. Then we made it into an Excel workbook and started rating each property we looked at on its own sheet. Both the exercise of making the lists, the side-by-side comparison, and frankly, the sheer number of tabs to show us there were always more properties out there, all were very helpful.