I love this idea - it utilizes both of your skills and allows you to keep the profits.
I wanted to say that first, before advising you to count the costs as to some issues.
A few thoughts on things to think through before you plow forward, though. I don't know the answers to these, but here are things I would personally consider in your position:
1. Can you go ahead and build two phases at once, thus increasing occupancy/profits enough so that leaving your job works sooner? Or keep your job through occupancy of phase one, until phase two? That way, you do not have the extra risk of losing your income.
2. How vibrant is your housing market? Others are right: this is high reward, but high risk. You'll be making a large bet on the growth of the local housing market and economy. In some places, that's a great plan. In others, it's risky. It's riskier the more your local economy relies upon one or two employers or industries to make it, or the higher-end your units are (thus pricing out more of the market).
3. Plan out contingencies: what if supplies / building is delayed? Can you carry the loan, or what are your options in that case? Can you refi the loan if rates go down? What can you do to mitigate known risks?
4. How would another covid situation impact you? You know your market, but I gather that it hit harder in places with longer eviction moratoriums. You might get stuck with occupied units for years in some places. You are probably already familiar, but be very aware of what costs/risks are involved, especially legally, in dealing with local landlord/tenant issues.
5. What impact, if any, will the increase in evictions now have on your market? Evictions were on hold in many places for years but are now going again, which is putting more rental units on the market once again. (It may be a minimal impact, but I don't know your proposed market/renter profile.)
Beyond that, I would get a good bit more advice from real estate pros before jumping fully into it.