Do your parents know your brother didn't repay the $400 loan?
If not, then you can let it "slip" when they keep badgering you. Stick to the "no, no, no" spiel. Eventually you can blurt out "If he can't even pay me back the $400 he owes, why the &#*@ would I loan him tens of thousands?" This is basically what you'd be doing if you cosign. I wouldn't bring this up right away, show that you're trying to be reasonable and not wanting to hurt other people's feelings. Hey, if they drop it right away, that's exactly what you'd be doing, right?
If they already know, you can bring it up right away, calmly and rationally.
Do you want them to keep paying the loans that they said they would? Say something like "I understand you're having financial issues, I'll gladly repay the student loans you took out for me." Don't want them to keep paying? Find out the details of the loan (easy if they're in your name and parents cosigned, not so easy if they're in your parents' name) and just pay it off. I'd go for the latter option, even if you do this you'll still be obligated to them, but at least not as much (not legally obligated, or even morally perhaps, but obligated to them in THEIR mind).
The advice to have the brother come to you to ask about cosigning his loan is good, but might not be appropriate in your situation. I believe your parents took out loans to cover your expenses, and they're paying for them? They are expecting you to repay that "debt", so I think they are the ones you should be dealing with.
As for the credit card, definitely get your dad off your account ASAP. As in, before you even posted here. Go do that NOW. Next step is, how do you break the news?
You can simply tell him the hard truth. He'll probably be upset that you didn't at least talk to him first, that you're treating him like a child, etc.
You could bend the truth a bit. Call your credit card company and tell them you lost your card and need a new one re-issued. Tell them that you first want to remove your dad as an authorized user, THEN request a replacement card. If his card has the same number as yours, then the order really doesn't matter. If it's a different number, DEFINITELY cancel him as an authorized user, right away. Now call your dad, explain that you lost your card and are having a replacement sent. In light of recent events, you've decided to not have the credit card company send him a replacement as well. He'll still be upset, but maybe not as much? He thinks you didn't call the credit card company specifically to remove him. Rather, you were getting a replacement card anyway, and decided not to have his replaced too. Plus, it easily explains why it was cancelled before talking to him about it.
Personally, I'd go with the latter option in this case. While honesty is usually the best policy, sometimes "well, TECHNICALLY I was mostly honest..." may be an even better policy.