I'll disagree with the posters above and say, hold onto your emergency fund.
Imagine the following scenario - another market crash and economic downturn resulting in you being unexpectedly out of work. That would be the WORST possible time to take your money out of the stock market, yet if that's where you put your emergency fund, you would have to sell at the bottom.
Emergency funds are especially important for one-income families, as you need to be able to cover 3-6 (or, my preference, 9-12) months of living expenses in a dire emergency. Yes, I know it's frustrating not to have that money earning more, and I'd suggest that you put some of it into shorter-term CDs or money market type accounts where you would access it without penalties if that happened.
Now, if you're comfortable with a 3 month emergency fund, put that money aside and invest the rest.
If you're interested in landlording, make sure to include costs of maintenance and an estimated vacancy rate. IF you can find a place where you only need 2/3 of the rent to make your mortgage payment (leaving 1/3 of the rent to put aside for vacancies and repairs) it MIGHT be a doable deal. Run the numbers though.
As for your wife - the good news is, with her quitting her job, your income might not go down at all! I don't know what you've been paying in child care, but the savings in taxes and childcare (and business clothes and travel and business lunches etc) will probably make up most of her missing salary.