You have to pretty much intentionally work to get in serious trouble with the IRS. Here's my horror story, to illustrate how easy it is to fix simple, honest mistakes:
One year, I forgot to report over $10,000 in income. It was for contract work in January and February. I didn't do any contract work for the rest of the year, so when I went to do my taxes over a year later, I just completely forgot about it. 1099 probably got lost in a pile of mail or something. Two years later, I'm getting letters sent from the IRS to my home. The thing is, I'm not living in my home at this point. I'm on a 6 month sabbatical in Europe. My house sitters are just collecting mail for us, and being good house sitters by not getting in to our business and seeing scary letters from the IRS.
I get home and find the letters, with the most recent informing me that the next one will be certified, putting the tax court process in play. OH crap! So I check my records, realize that oops, I didn't report that income. I call the IRS number to make sure they're not sending agents after me. I tell the IRS worker that I made a mistake, explain what happened, tell her that I've been out of the country and I just got home to the letters, and that I needed to pay them today to get everything squared away. Her response: she really wanted me to go back and just make sure I couldn't deduct some expenses from that income, because she was sure I could. No "pay now!" type of scare stuff you might get with a creditor. I knew I had some tiny deductions, but I also knew that the time and effort to deal with it wasn't worth it, so I just paid what the IRS calculated. Total penalty was about $50.
So, I failed to report over $10k in income. I owed Obama about $2,000 in taxes for two years. I then ignored half a year's worth of letters from the IRS, until they were ready to begin the long, arduous process of the court system. And when I finally called, they wanted me to make sure that I took advantage of every available avenue to pay them less. Penalties and interest on that $10k mistake amounted to about $150 total.
When you read horror stories about the IRS, you are probably not hearing the whole story. It takes some serious, intentional mischief to get the IRS to come get you.