I see both sides to this. The implication is likely that children treat a baby under 12 months differently than they would a child their own age. In other words, they would rough-house with one and not the other. If he flung sand directly in my baby's face, I would likely be upset as well, though I wouldn't have confronted you about it. Then again, what is the age range on the playground? Likely three years and up. As a parent of older kids (plus a two year old), I do get frustrated by parents of younger kids who expect the older kids who are playing on age-appropriate equipment to be careful around a toddler who isn't playing on age appropriate equipment.
When you only have a baby, older children seem very big to you. Now that I have an eight year old kid, he seems downright tiny to me, but when he was little, a kid his age seem rather threatening on the playground. I probably incorrectly assumed that children who were just being children were being rambunctious.
Once I saw a confrontation on a playground similar to yours, and I ended up sympathizing more with the parents of the child who had been wronged. The reason? Because the parents of the kid who was hurting other kids were very defensive and refused to take responsibility for their child's behavior. Now I knew the violent child has an IEP at school and lots of behavioral issues, likely autism spectrum. But that doesn't excuse his violent behavior, and it made me sad to overhear the parents of the violent boy say, "I hate it when parents intervene." He was pushing kids to the ground over and over again, and slapping them.
But if you apologized and did all the necessary things to make the situation right, then yes, they were overreacting, though I understand how hard it is when you only have one child. If it were me, I wouldn't have implied that he was overreacting. Did you directly apologize for your son's behavior as well? That might have helped the situation.