Definitely do not cancel until you've replaced it with another card, which includes downgrading, or raised the credit limit on another card, else your credit score will suffer. Utilization is important. So is the age of the oldest account.
Barclaycard waived its fee for us on the AAviator card for 2 years but not this year; DH, DS, & I each had one. Since DS only has a few credit cards, he was able to get the credit limit raised on one of his no fee reward cards, then cancelled his Aviator. If he finds he needs an AA card later for the free checked bags, better boarding group, & 2x AA miles, then he can apply for the Citi version & get its bonus miles. We simply cancelled ours because we have cards with a total very large limit & are lifetime Platinum members. Several years ago we downgraded our Citi AAdvantage cards to no fee Citi Double Cash cards which pay 2x on everything, but Citi also agreed to 'age' the cards the same as the AAdvantage cards we were cancelling, which meant our credit score wouldn't be hit. The Double Cash card is the one we now use the most but although it calls itself World Elite, it (and the Citi AAdvantage card) charges foreign transaction fees. Barclaycard, BOA, Chase, & CapitalOne do not.