I used to want 4 kids, and my wife wanted 2, so we tentatively decided to have 3. It took a few years for our first one to come along (he was born 2 weeks ago!). I'm now 32, and I no longer want 4 kids. Because it took so long for our first kiddo to show up, I spent my time learning new hobbies, reading, and just enjoying my freedom. Now that little Manly Father is here, we'll be thinking long and hard about having a second.
He takes a lot of time, but he is worth it. When a second child enters the picture (or 3rd, 4th, etc.), it is no longer just your time you are missing out on. Your older kids get less time with you, and this can be challenging for them (given their lack of impulse control/emotional control).
Having 1 kid = relatively easy. Having more than 1 is not a linear increase in difficulty: it is an exponential difficulty increase you won't fully appreciate until you have kid 3.
If you are trying to logically decide on paper why having 3 or more makes sense; it won't. This is where your biology kicks in and says whether or not more are "worth it." On paper, I shouldn't be as happy with my baby (he takes up all of my time, money, and energy and pays little in the way of dividends), but my biology pumps out massive doses of oxytocin and dopamine whenever I look at him. These biochemical highs are what makes it awesome for me.
Before taking advice from someone else, who has a completely different frame of reference and a completely different set of experiences, you need to do some deep introspection. What do you want out of life? What is missing from your life? How will you get what you want and replace what is missing? Does adding another human to your family mix help or harm your (and your partner's) long term goals?