We'll, that's the thing. I don't pay anything for housing right now. I have an incredibly lucky situation- I don't own the place, but live there rent/utilities free. So no matter what I do, or how much I pay for it, it will be a 100% increase.
If I buy/build, that potentially means I need to work longer to replace the money I'd spend for that. Right now, this living situation does not have any kind of expected end date other than I'd like my own place (the dream being I create a permaculture homestead- started one before but ended up getting divorced and moving for my current job).
Of course, the situation could unexpectedly change at any time, and I guess that's another factor in me wanting my own place. I like the security of having my own place, producing much of my own food, etc. I think my "middle ground" would be to see if I can get the piece of property I'd like to have, but hold off on building until I've got a little extra padding. Then I could just plan on working a few more years, or part time for a while to cover some of my expenses to let the savings grow longer.