You are definitely not nuts to start camping with the little ones. The first couple of times may be a little tough (depending upon your kids' temperaments), but they'll quickly get used to it and love it. Plus, camping can always be tough the first couple of times, no matter your age, best to get them started young. My husband, and toddler (he was two at the time) spent nine months camping around the US in a 3 person backpacking tent. I should mention - we were car camping the whole time, not backpacking. It was awesome and he did so, so well. We never had a problem with him getting out of the tent, but we put him in between us and we were in a small tent, so we definitely would have woken up had he tried to get out. We bought a kid sized sleeping bag for him that he can grow into through his adolescence. To make it smaller, we could just stuff the bottom of it in the stuff sack and cinch it tight. Your guys could use backpacking (i.e., shorter) groundpads since they're still short and their whole body will fit on it.
The biggest things we had to adjust to was going to sleep while it was still light out and sleeping in a sleeping bag. The light problem was more pronounced in the middle of the summer (duh) and did mean that our little guy had a harder time getting to sleep and woke up with the sun. If you've acclimated your kids to really dark rooms with blackout curtains as we did, you may find this a problem, as well. As for adjusting to the sleeping bag, I just went to "sleep" with him at the same time he did for a few nights to help him get the hang of it, and eventually he did. You could try camping in the living room at first to get them acclimated to the tent and sleeping situations without the other camping variables.
Lastly, finding relatively quiet areas to camp was a huge plus for us. Especially on weekends in the summer. Loud, drunk neighboring campers never made an easy night for any of us. We got pretty good and predicting the types of places that would attract loud partiers and tried to avoid (we found fewer loud partiers in national parks), but you can never completely predict who your neighbors are going to be.
One of the things that I thought would be a challenge was getting him to stay in our site while we were setting it up. Turned out to be a non-issue. The very first night we camped, we asked him to collect downed sticks for a fire. He was like a machine - I could not believe he collected so many sticks. He continued to do this night after night - though this is only useful if you're camping somewhere where you're allowed to collect downed wood and where there is downed wood. We brought very few toys for our adventure, and I found that one yogurt container full of mini animals was enough to keep him entertained most days when we were setting up. He loved setting them up all over the campsite. We also got him involved in the process, which was a huge win all around. He loved it, we weren't worried about him getting out of site, and we had help, to boot!
Most kids love to be outdoors, so there really isn't much you need to bring for them other than what they need to sleep, eat, poop, etc. The two camping gear items we had for him that he loved were a camping chair of his own and his own headlamp. The latter was such a good thing to have, both for safety reasons and for entertainment purposes.
Best of luck, and most importantly, have fun!! If I think of anything else, I'll chime back in. But if you have any specific questions, feel free to send me a message.
