Also, have trouble with anything that starts so late as to almost definitely not be something observed to be true by people long ago, but by some random guy like Smith who says "let's go with LDS!"
Wait, so this bothers you, but some random guy like Jesus who says "let's go with me!" doesn't? What is it about that time period that makes it seem to you like there would be a higher standard for truth? The reason we know Smith was full of it isn't because we are applying the same reasoning as some bronze or iron age civilization.
Anyway, you wanted downsides of belief. Well, if you want to believe in some nebulous concept of a god, go for it, but you hobble your own ability to understand the world. There's always the cop out of "god did it" when you don't know, and never an incentive to look past that. Gods were a great stand in for "I don't know" and humanity loved creating them for concepts they couldn't quite grasp as wide ranging as thunder, lightning, the sea, the forests, and everything else. We've moved beyond a lot of these as we figured out stuff like electricity is a thing and that there wasn't a god up in the clouds tossing lightning around. We've also stripped that power from the monotheistic gods, despite the clergy's protest that things like lightning rods were subverting god's divine will. Apparently enough churches being smote while other buildings were safe will get even the clergy past that mistaken belief, so the downside of that particular bit of ignorance won't likely affect you. Of course, you'll have to ask yourself what physical phenomena you might be willing to hand wave away. Maybe you think some god won't let our planet become an inhospitable hell? Certainly enough of those to go around that it could come out to be a downside.
There are those parents that think some god is going to spare their child, so they pass up actual medical help. A number of people that chose faith "healing" and neglected to get proper care for their children have gone to jail for the child's subsequent death. I'm not sure jail is appropriate for those folks. They're delusional, sure, but didn't seem intent on killing. Anyhow, certainly a downside of strong belief. Maybe you're more open to keeping god out of everyday life and will leave your god to the matters not involving the physical world.
If you want to formalize that belief into a religion, like so many seem to, then there are myriad more downsides. The number of people on this forum who feel compelled to hand the church 10% of their income seems like a pretty big downside. Maybe you'll pass the offering plate by and this won't be a downside for you, though then being ostracized by the rest of the congregation might be. If you do pay up, hopefully most of the money will go towards some good and not buying somebody a fancy car or a jet.
Religion has been a great historical tool for those in power, getting the believers behind political causes. Great proxy for the UK and Ireland, where people were being killed based on being Protestant or Catholic because it was so close to defining the sides. But the troubles have settled down, so no downside for you there. Crusades aren't modern so you're safe there. Giving the public an outside religious group to scapegoat so they'll go along with otherwise questionable political acts defies time though, and there is no shortage of it today. Could be a downside for you if you get swept up in it. Could be a downside for all of us if enough others will trade away our collective prosperity to persecute the outside group.
There are some big downsides for people that grow up religious and turn out to be gay. No shortage of problems for people that can't reconcile their upbringing telling them they're an abomination for their biological bent. No shortage of families torn apart for the same reason. But statistically you're probably not gay and probably won't have gay children, so for you individually it won't likely be a downside. And I'm sure you can separate your faith from your treatment of others, so you'll never experience the downside of not getting to know somebody because their actions conflict with your belief.
Religion provided great justification for slavery, and I'm pretty sure many of us are dealing with some of the downsides of that yet today. But maybe you live in a homogeneous community of people not adversely affected by that and haven't had to deal with any downsides.
There are plenty more downsides, but hopefully that's enough to provide an existence proof for you. Like with many things that change the balance of power, there are upsides for people as well. Maybe you're lucky enough to not be negatively affected by the touch of religion. Maybe you can call yourself faithful while picking and choosing only the bits you want so you won't fall victim to some of the downsides. I don't know, but it seems pretty silly to pretend that there are no downsides beyond just being wrong.