I see a lot of new business owners let their bookkeeping slide, then become overwhelmed at tax season - then they either get their taxes done in a rush, or they neglect to file.
So my suggestion, even for very small single-member LLCs, is to open yourself a no-fee small-business checking (shop around, you should be able to find one at a local bank or credit union), or even personal checking account, and keep your business income and expenses separate. If you use a credit card, have a separate credit card that you use only for your business (as a single-member LLC, it's fine for the credit card to be in your name instead of the business').
Then do your books on a monthly basis. You have the advantage of knowing how you did in the previous month - what worked? what didn't? who owes you money? - and being able to use that to improve your business. And, your books will be done in January, with no stress or fuss for tax season. It will also give you a jump-off point to calculate estimated taxes as you go.
As a single-member LLC, the IRS doesn't require you to have a separate bank account or credit card, but it will make your life easier. If you ever fall behind on bookkeeping, you know that you can just pull your statements and get a complete picture - no rummaging through personal accounts, trying to remember if something is a business expense or not. It also allows you to download all transactions at once and automate your bookkeeping.
You can use a bookkeeping software or Excel - here are the advantages and disadvantages:
1. Excel - Free (if you own it, or just use Google's version), but kind of clunky for bookkeeping with very little capacity for automation or reports.
2. Wave - A free cloud-based accounting software that will let you link your bank account/credit card. But it doesn't allow you to create automatic classification rules, which is essential to making bookkeeping efficient. Supposedly it eventually learns, but I've never had the patience to get that far. It's a small step up from Excel - but a step up, and still free.
3. Quickbooks - You can do Quickbooks online with a subscription fee or Quickbooks desktop with a disk/download. I favor Quickbooks Online - and so does Intuit - it's clear in how they direct their support and development dollars that they believe that Quickbooks Online is the future. You can get a subscription for $10/mo or less - be careful not to buy too much subscription, because though you can always upgrade your subscription, you cannot downgrade without restarting your company books!
4. Xero - Subscription based cloud-based accounting. Directly competes with Quickbooks. Works best if you direct link your bank and credit card, but overall it's similar to Quickbooks. It's more popular internationally than in the States, and as a result you might find that local accountants aren't familiar with it. But maybe that's a good thing - you can just weed out any accountant who is unwilling/unable to work with Xero, because if they're overwhelmed by accounting software, they probably suck.