In my opinion, not yet. Assuming you use the equity in your home as your down payment without more, you only have a 12.5% down payment, meaning you will have to pay for mortgage insurance. You paid off $30k in student loans in the past year. Save that much again this year and you will have 20% to put down (assuming you don't lose equity in your home). Let this house go. There will be other beautiful houses that suit your needs available when you are more ready to buy.
The payment on a 30-year loan for 320,000 at 4.5% is about $1,620.00, or approximately 15% of your income. Initially, $1,200 per month of that is interest. Factor in real estate taxes and insurance and you are easily throwing away at least $1,500 per month of money you will never see again and spending closer to 20% of your income on housing. Keep in mind that is a conservative estimate and ignores things like maintenance, utilities, etc.
Without more information we don't know how a $400,000 home compares to other homes in your area or the cost of renting, so it is hard to determine whether this would be a reasonably necessary expense. But assuming there are less costly options out there, you would likely be better served to look at less expensive housing options.
One thing I think people should always consider is whether you would be able to live on the home if one of the two of you were to stop earning money. I'm thinking you would be hard pressed to afford the house on one income unless one of you significantly out earns the other. And in that case you can only afford it if it is the lower-earning spouse who quits contributing income.
Of course it is ultimately your decision to make based on your values, preferences, and risk tolerances.