I was in a similar situation and DID cancel.
My score is in a similar range to yours, and this was by far my oldest card (didn't use credit cards at all until I was in my early 30's, and had just that one (with a limit of $400) for years, before I took my current job which has a ton of paying upfront and getting reimbursed to it. This card was four years older than all the rest of my cards.
The fee was $79 per year. On a $400 credit line. Which I never used. So what ended up happening every year is that I would set a reminder to pay the fee, just to keep the 'length of score' part of my credit score in the green. It was due right around the holidays, so every year I would forget, get increasingly panicked letters sent to old addresses (I move a lot for work) until one caught up with me, and then call and ask them to waive the fee, which they would do. I never paid the fee, but I also never used to stupid card.
This year I finally decided that this was frankly the stupidest thing I could possibly imagine doing and there had to be a better way. I called Capital One. Can they take the fee away from the card? No. Even though I've never paid it in the life of the card? No. Can I talk to a supervisor? Yes. Can she permanently waive the fee? No. Can I switch to a different card that is more in line with my spending patterns. No, because they have no way of knowing my spending patterns because I never use their card. Fair enough; can they increase my credit line to a point where there would actually be some logic to using the card? (I was looking for a new emergency card, and thought this one would be a good bet because they keep it open even though I never use it... unlike my last card, where I discovered in an emergency that they put a 'hold' on the account if you don't use it once a month). After having the card for 10 years, with a credit score in the high 700s, they offered to increase my credit limit to $500.
Flushed! According to the tracker, my score went down between 7-11 points, but damn is it nice not to have to worry about that stupid-ass fee.