I'm not convinced we could ever go to a true cashless society. Money comes from other people, not the government. The government wants you to think it comes from them so they can control it, and get their taxes. If the federal gov got rid of cash, I would think another physical "underground' or local currency would pop up to take its place. Or people would hoard dollars and trade those for as long as the cash would hold up.
The one thing I don't like about a cashless society is 1), grid goes down, then cash is king. What are you going to trade? this is why I actually keep some money in the house and 2) in theory the gov/companies can control your purchases. I know it sounds extreme, but could there be a scenario where if you're tagged as rich and there is a shortage of something, could the gov force the CC company to decline your transaction. Sounds extreme, I know, but cash is the only true way to give yourself full control of what you purchase.
Being tracked by your purchases is also another concern, but we have that today already.