At about 4 or 5, I had a swig of curdled milk and after that I immediately stopped drinking milk. Like, I specifically remember that incident as a turning point. I LOVED milk before that, too. To this day, I can't really stomach it. I went years eating dry cereal in the morning before discovering that almondmilk was alright.
That was over 20 years ago and I have never broken a bone or had any issues with calcium/vitamin D deficiencies. I never really ate extra dairy to compensate, either.
My point is, like others said, kids are strange and fickle and if they're okay with it (and it's not something that will kill them), then you can probably be okay with it.