I think you wait until they're in and the doors are closed.
That goes double when there's kids. Kids are unpredictable and you know the parent is distracted. There have been several times when I've seen a parent trying to load one kid and then another kid starts getting out of the car, or runs around the car.
Sure, maybe I can see that it's safe to back out around them, but the parent can't, because they have to have their attention on the kids, so it's rude to make them worry about what you're doing behind them. A half foot of space isn't a lot of space for a person to feel safe with a vehicle moving right next to them. The driver might feel fine about it, but they're not the vulnerable one. I wouldn't move my vehicle past anyone with only a half foot of space - not a cyclist, not a pedestrian, not someone changing a tire. It's not different in a parking lot.
I will also add, that I find it distracting to try to pay attention to what they are doing, check to see if all their kids are corralled, and know what's happening behind me. Why take the risk? No one is in that much of a hurry.
The only exception is if they make eye contact and stand there with the door half closed, trying to let you go. It then becomes awkward to not back out, though I personally would prefer that they would just finish and get into their car.