+1 to doing your due diligence. You want to make sure you can get electric, water, sewer or septic, the whole 9 yards. Don't buy something only to realize there's a problem later when you go to build. This is much easier if the property is in town because odds are many of those things are a given but you should still investigate.
If your loan would be seller financed, meaning no bank involved, land is the easiest real estate transaction you can make. A written contract is all you need, laying out the terms of the loan, and you take that to a title attorney for closing. The deposit is a bit trickier without a real estate agent. An RE agent will hold that deposit in escrow, making it easier for you to get your money back if you would bail during the contingency period. Not sure if a title attorney would do that for you. If you give the deposit directly to the seller, you bail, and they don't pay you back, you'd have to take legal action to get your money back.