Yes, one of the minor downsides of this is your address may show up on their credit report, meaning:
1) You may get junk mail for former AUs at some point. Some people have reported this, though it doesn't seem super common. I haven't had it happen, because I use a virtual mailbox for all my mail, and they only process/deliver mail addressed to me or my wife, and reject anything addressed to other names, so I never even see that if it is sent to me. The remedy, if you don't want to get a separate box for AU sales, is to just throw out the junk mail if you get it.
2) You may get phone calls from creditors for that person, if the creditor runs their credit report, sees your address, googles it, and can find your phone number. Again, it's happened to a number of people, but doesn't seem super common. Again, I haven't had it happen to me because my number is not super googleable, but the remedy, if it happens to you, is to ask them to remove your number and/or block their number.
Annoyances, yes. How big or small probably depends on your tolerance for such things, how frequently they happen, and how much you got paid to put up with it.