No, stick with what you've got. Someone recently told me that families in smaller homes are happier. You and your wife can get clever with the way you utilize the space, you could make it a beautiful haven for your family.
Big homes aren't usually worth it. More space to heat/cool (probably not huge in SF, but still), more rooms to clean, toilets to scrub. More places for kids to hide away from their parents.
I live in about 1,300 sqft with my family of 4, and really don't want much bigger. It's a 3 bd, 1.5 bath, but my kids share a room and the 3rd bedroom is small and rarely used. The half bath is in a section of the house that is poorly insulated and can get to freezing in the wintertime, so it's really only seasonally used. So really, we use 2 bd, 1 bath most of the time. We rent, and I have a lot to say about the layout, but generally, it works well for us.