I'd probably buy the lot.
My sister had the opportunity to buy the lot behind theirs when they were having their house built, back in the mid-nineties. They didn't buy it, thinking a fence would be fine for privacy. The people who built behind them put in a deck, and when they sat out there they were looking right over the fence into my sister's house. It took a long time for trees to grow and screen them off (Canada, short growing season).
Also, the lot would have been worth way more than the selling price within a few years.