I always thought, if it's not moldy, it's not going to kill you.
But I have had some funkyness come out before. Your temps are pretty much what I use. Though I tend to push the envelope a bit and add my starter around 48-49. I've been making mine in the Instant Pot lately, so incubating is a bit more consistent, but when I did it in a crockpot it would occasionally get off. I think it was because it got to cold and didn't culture properly. Instead of leaving it on the counter, maybe wrap in a towel and leave it in the oven with the light on to slightly warm it.