Human nature to cast doubts, always a bitter pill to swallow. Human nature also, to seek validation. :)
That aside, OP, have you read/encountered Jacob's blog, Early Retirement Extreme? I find his voice more mature than most early retirement/personal finance writers - MMM included. May want to go have a peek if you are interested. I believe he considered himself "retired" and was able to live off of $7,000 per year (his wife works/worked, and the $7,000 includes only his share of the expenses).
At a 3.5-4% safe withdrawal rate, you need about $200,000 saved/invested to theoretically live off of it forever, to draw $7k to 8k per year. I believe that's exactly what Jacob did.
Housing is just one piece of that puzzle. You indicate that you would homestead and grow your own food, that will help lower your annual expenses. But basically, if you haven't encountered this "advice" yet, you need 25-30 times of your annual income invested before you can consider yourself retired. If you don't add to your 95k capital in the bank, you can invest it and theoretically withdraw $3300 per year, before tax and not deplete that 95k during your lifetime. With your house paid off and assuming $300 per year that you would need to maintain your garden (seeds, supplies and the like, assuming you are able to get your own rich soil by composting and minimal watering needs in Portland), you have about $ 3000 left out each year. Is that going to be sufficient is a question that you would have to address/answer. But by the time you turn 55, if you have, say, 200,000 invested, you can draw may be 7000 each year and not deplete that fund.