I don't think this is a question that anyone other than you and your wife can answer. My husband and I have one so far (9 months) and we're still in the midst of the "OMG, this is rough some days" period, but it's getting much easier. My "me-time" went down a lot (as the mom) in the beginning but it's picking back up again. I have time to read, time (and inclination) to exercise, I managed a small garden this summer, I've still been preserving and cooking and foraging, and now that it's getting into autumn I'm taking on some knitting projects. Free time will never be what it was pre-child, at least not until she's out on her own, but there are ways to make time for yourself and for time with just you and your partner. All this, and ours is a relatively high-needs baby. Not so much as one with developmental problems, but demanding a lot of attention and focus. My husband and I are pretty good about letting each other know our needs. "Hey, I need to get this done. If you watch her for half an hour, I'll take over after that and you can spend some time to yourself." She doesn't always let this plan go through (sometimes she just wants Mom) but like I said, it's getting way easier.
We're planning to move closer to family next summer and I think that will make things even easier on us and, likely, better for her since she'll get so many more people to pay attention to her.
We'll have one more, but just one. Originally I wanted 3, husband has always been set on 2. It's a complicated math I've done with myself over how many kids to have. If I was thinking solely of the environmental impacts, 0. Only about how much I love kids and babies, 6. Only about how much I hated being pregnant, 1. Based on finances alone, 1-2. Based on my need for time to myself, time for hobbies, etc., 3 (under the assumption that they will get more independent as they get older--which my daughter is quickly proving to be true). If I'd known how awful her birth would be, maybe 0-1. (Maybe.) If we had more family close-by, ready and willing to help out during the first year, 3-4.
See? Complicated math. As someone else said, wait until after you've had the first one and then see how you and your wife feel. I had a relatively easy pregnancy and still, hated it. Her birth could be used as a campfire story to scare people. And still? I'd do it all over again. All the sleepless nights (like last night), the discomforts and pain, the (very few) moments when I've thought, "I really can't do this, oh God I really wasn't actually cut out to be a mom what have I done??!!", which were probably fueled mostly by exhaustion, the worries which start immediately, etc. This has been more worthwhile than anything else I have ever done in my entire life and I love it.
My husband, while wanting kids "someday", didn't want them as soon as I did. I'm 2 years older, so age played into it a little bit. Turning 30 really made me realize that I wanted to finally do some of the "adult" things I'd pushed off during my 20s, like having kids. Obviously, I got my way in this. How does my husband feel? He wouldn't trade her for the world either. Other than going to school he's SAHD so it's a huge change for him and one I don't think he ever anticipated, but he's loving it. He actually had class last night during her bedtime and when he came home he said, "Man, a big part of me really wants to wake her up just to say hi! I missed her." I think it's helped to actually know that she hasn't made him completely housebound, or stopped him from doing his favorite things, and the older she gets the more fun she is because she can do so much more month by month. It's incredible to watch.
That being said, I would like to think that if we were infertile, I could accept it and wouldn't put us through the pain and hassle of IVF. I already know that my husband wouldn't have been up for adoption. I might have been able to talk him into fostering kids (and am keeping it in the back of my mind later on), but we might very well have ended up childless if we couldn't conceive naturally.
There will be people on these forums who hate me for this, but I kind of wonder if having a difficult (but developmentally normal) child is correlated with higher intelligence? It might be one reason why smarter people have fewer kids: more difficult children.