Totally depends on who its for and why. Sometimes you need to not be *that guy* and just chip in $10. Other times, if it is someone you do not know very well or do not like, its OK to say no.
Way back in the day, the IT team worked on a higher floor than the rest of the business. We were secluded and enjoyed it. However, once every few months, a downstairs person would show demanding money for something, just like you are telling us. Each time everyone on the team would talk up how they are not giving any more, blah blah blah. Each time they would change their tune and fork over their hard earned money to someone they did not even know whom they never worked with. It was quite comical.
There was one exception to the rule, me. I decline or was not even asked. It was never a big deal. I did not get fired, or ostracized, or anything else. In fact, it made me more popular among my fellow IT workers for standing up to "the man".