How hot does your area get? The only downside I really see to a colder house in winter (as long as it's warm enough not to risk freezing pipes and things), is that heat in the summer is going to be a lot harder to handle, and it can be harder to stay cool than stay warm.
I live where temps can be (converting to C...) -22C in winter, to 47C in summer (-8-ish to 117ish°F). I've gotten used to a cooler house in the winter, but "I don't do heat" ever since I was little (I'd faint a lot if I was outside, now I've learned to see the signs and sit down to avoid fainting). That's made summers even harder for me the last couple years.
Without those extremes, you're probably fine having the house chilly. When my son was little, the guideline for being out in the weather was "add one more layer than you wear as an adult". People tended to way overdress the babies when it was cold, thinking they couldn't handle it. So, if you're home, they can wear pjs, sweats, socks, etc. Pretty much what the adults wear, with a onesie underneath for the tiny folk.