Just posting to say you have my sympathy.
You have sold this property now? How does this guy even know how to contact you?
Last house I bought I had one neighbor in the back yard who always seemed to have a complaint and always wanted me to pay half for his suggested various projects. One of those things was a fence. My back yard was a wooded area, bordered 6 other properties in the back, and there was a small stream running diagonally through part of it that would be impossible to fence. I had no interest in paying anything to have part of his fence put along about 1/3 of one edge of my back yard. But he brought it up several times. It got to the point that I dreaded seeing him when I went out there. So, having dealt with a sort of similar situation, I'm going to say that any exasperated "fine, build your fence" response you gave him is not the same thing as saying "sure, I will pay for half of your fence, let me know how much I owe you."
Glad you are getting some legal advice.