Kelstache, I'm so excited for you.
We just went through a long process of trying to buy land, then do a major home renovation, and now are beginning to look at building the farm.
You've got some great advice here, but most importantly, it doesn't sound like anything that's been raised is something you haven't thought of. You have a strong financial and family base; you are planning to go slow; husband is happy working; you have the *option* of going back to work.
The financial picture is tight, but not impossible. Personally, I would say go ahead, because it's the dream and if you are planning to be there a long time, and if husband and farm income can both go up over time, this should be the most challenging time. If you can get through these next years intact, life should get easier and this will have been a solid plan.
I would think and plan carefully though about back-up plans. We did this extensively and it helps me sleep at night!
*make sure you have lots of life insurance for DH as sole breadwinner
*make sure you can afford all of your costs to live if the farm doesn't get up and running (ie you have to buy all your food, pay for heat, etc. We're having to do all of these things right now as it takes time to get other systems up and running)
*make sure your home will have reasonable resale value. IE don't make it SO personal that it will be difficult to sell or too expensive for your area.
Your biggest back-up plan if you decide you can't or don't want to make a go of this will be to sell. What you're taking on is a LOT, even going slow. If one of you gets sick, or you end up with more children or a special needs child, or just burn out, or something else comes up that becomes a bigger priority than farming, you may need to move sooner than expected (I have seen all of these happen to young farmers in my area). Make sure that what you're putting most of your money into remains an *asset* financially.
*have ample funds for farm infrastructure and set up. Know the labour costs in your area for renting excavators, building fencing, etc. $5000/year is ample for setting up gardens and maybe a greenhouse. Fencing our two acres alone will cost $5000 for us to do and would cost $15000 to hire someone else. Many things can be done cheaply or for free in the short term (as I'm sure you know), but cheap and free falls apart and has maintenance costs and the short term comes to an end at some point. I would encourage you to create a farm plan that envisions what you want to do, the time and labour involved, the costs, and how LONG it will take to implement, realistically. This includes working toward farm status. Time becomes a limiting factor very quickly, IME, and then money becomes an issue.
It sounds like you are on top of all of these things and moving cautiously. With that attitude, the odds are in your favour, and it's likely worth the risk. But back-up plans are really important.
On an aside, I'm curious about your decision not to keep animals. I get not wanting to take care of more creatures with baby on the way! :) But animals are often far more profitable, less time/labour-intensive, and create a faster path to farm status out here. Annual crops are serious labour for lower pay, often. Just curious! :)