I think it is fine to say that you don't want to join in on the gift exchange. When I decline, I always joke that I come to work to make money, not to spend it! After last year's white elephant game, everyone traded to get back the item they bought. What a waste of time. If they think it is fun, fine, but I am not interested. But don't get me started about the year that the "Office Christmas Party" was to go out to lunch, paying our own way, to a place that gives a hefty discount to everyone except two of us in the group (non military). Wow. And a happy Christmas to you, too, Jerk.