I've thought about this scenario for my family, but, so far, haven't done it.
My big question...what does your significant other/family think about it?
Another question: If the goal is to do it in order to pay off the house faster, why not re-consider needing the house at all? Meaning that if you're comfortable living in a small shed for 5 years (which is a pretty long time), why not just make that a permanent lifestyle plan? What would change in 5 years? There are lots of people who have gone down that road (just living in a tiny house from the start)
If you're the type of person/family that would be happy living in a shed for 5 years, why not just dump the modern idea of a house, skip paying off the mortgage (sell it or just rent the whole property) and adopt an "alternative" lifestyle (as in living in an RV/travel trailer, boat, tiny house, small studio apartments, etc.) without plans to move back into the house.