Yeah, look at form 8880, I believe. Any withdrawals are subtracted from new contributions before figuring the credit on the balance, if any is left. I just noticed this a few months ago - it completely scuttled a plan I had for a slide to early retirement. I had wanted to have DH quit work while I continued part time. The thought was begin Roth conversions from tIRA, I make Roth IRA contributions for both of us from part time earnings enough to capture Saver's credit to offset taxes on conversion, and withdraw past Roth contributions to cover remaining expenses. Unfortunately it won't work.
Even uglier - any withdrawals are included in total on both spouses' sides of the equation, and they include I believe all withdrawals from the past 3 (I believe) years! So a withdrawal can follow you for a while, making you ineligible for the credit for years.