Lots of good answers here!
I (male) agree with the premise that if DW says no, the answer is no, but if I say no, it's (somewhat) negotiable.
For us, it was pretty easy. Actually, that's not right. We struggled a fair bit to make up our minds, but we were on the same page throughout (which made it "easy"). We talked about having kids before getting married, and we were both ambivalent. Over time, this continued. I think I was more no, and DW was more truly ambivalent. Personally, my no was largely coming from fear -- the world was so messed up, how could I bring another kid into it? I was also afraid I was too selfish. We were prepping the nest (ie buying a house, putting down community roots, holding stable jobs, etc.) but at the same time I think we were leaning towards not having kids. Then we joined a local Buddhist community. Once we no longer felt that we were all alone, swimming upstream, we decided to have a kid. DW got pregnant right away. Birth was smooth, so we could have a second, but I think we'll just have the one. DD is 2.5 and DW just went back to work.
We're not young (I'm 41, DW is 37). I also feel like 1 makes travelling easier. DW is an only child, also, so it feels right to her. I can't say 100% that it will just be 1, though...
If we were younger, I could definitely see having a big family. Being a parent is awesome. :) It's amazing to me how unconflicted I've felt. When we left the hospital, one of the nurses told me: remember dad, it's the baby's watch, the baby's calendar, and the baby's wallet. That was extremely helpful, and (for selfish me) surprisingly easy to accept.