Southern CA with a 4 mile commute on what sounds like a reasonably safe route sounds ideal! I would definitely be a one car family in your situation. We are in New England with a baby and a toddler and have one car, and we're in a town/suburb, not in a big city where that might be expected. So it can certainly be done. Though I have to say this winter was awful and almost broke us, due to partly to unusually snowy weather (bike path that my husband usually takes was unpassable for much longer than normal) and partly because I started a job in January that required a longish car commute a few days a week (before that I had a crazy long commute one day a week but it was all on public transit, so he always had the car as a backup). But, I've just taken a new, semi-retirement job at the same, bikable place as the husband, so we definitely won't be getting a second car anytime soon now.
Some things that help our situation:
We're really into cycling (or rather were before the kids), are both pretty committed to it, and are experienced (spent years riding for sport and commute, so not much phases us and we don't think biking with a young child in tow is crazy)
We very deliberately chose were we live to be bikeable
We have a nanny, so the toddler has to be taken to preschool some days a week but we don't have to pack up the baby. This seems like a crazy 1% expense, but the cost of nanny <= two kids in day care+ second car expenses. We could still do it with one car and day care, but I think it would be harder.