I carry cash because if things go wrong (power outage, Visa outage, natural disaster, etc), cash is still a useful means of transferring value.
Also because I'm concerned that cash is going to be banned in my lifetime, and I'd rather help hold that off as long as possible.
There's nothing wrong with carrying a useful amount of cash on you - if you don't want the temptation to spend it, wedge it out of the way in your backpack/cycling bags/etc, so it's not "always there," but you have it if you need it.
My normal problem is that I don't have small enough bills for "loaning someone $20."