More or less: this is exactly what we did.
We bought a bit of land near where our parents live back in about 2005. It had a 30x40 workshop on it with a very minimalist 600 sqft apartment inside. We used it as a weekender house. In 2006, we sold most of what big items we had, quit our jobs and leaped. We lived very minimally from 2006-2009, as it took that long to find reasonable employment. (My skills are relatively niche skills that don't really exist in small towns.) When our house sold, we owned the land outright with a bit left over.
We lived in the tool shed from 2006-2012. We started building a house in March 2011... and it completed right around the first of 2012. We had enough stashed away to just pay cash and that's what we did. We did a reasonable amount of the work ourselves, which made the build take longer. We also didn't complete the house 100%. (We left the upstairs unfinished.)
FIRE was in July 2015. The house is probably 85% completed. There is one room that is still not complete. We've started fostering cats, so... the cats live in that room. It will probably get a little bit completed between each batch of fosters, but the fostering has definitely slowed the process.