I'm glad I posted this here. I've read everyone's post, and I can't respond to them all, but you all gave me a lot to think about.
Actually, we have a 4th of July get-together set for tomorrow, so I've decided to still go, but to really be super mindful of how the interactions go tomorrow. Wives will be around, but basically it will be "wives in one group of chairs chatting" and "husbands in another group of chairs chatting." My wife and I are the only people who ever really walk back and forth and visit with both groups. Otherwise, it's pretty segregated. I'll see how good/bad it is tomorrow and make a decision from there.
The complicating factor is that the group is like "a package deal." Again, about 25 years of history. We all live close together. We do fantasy football together. We have this text chain that goes back forever, and lights up pretty much daily with random chatter. It would be very hard to "divorce" one of them without divorcing all of them.
The players:
B - single and the most outspoken of the bunch. Very bitter about life after some bad decisions. Spews the most objectionable bile. Frustratingly, can be very pleasant in some situations. We used to have great conversations, but that was 15-20 years ago.
D - married and very unhappy. Two small kids, which I think exacerbates the stress. Has been very bitter and negative lately. I would call him my best friend, and one on one, we have good respectful talks. However, he tends to echo the others' negative vibes when we get together.
M - married and very unhappy. Fairly progressive on some issues, and I agree with him on more than I do with B or D. However, he uses some racial terms I would not use, and tends to encourage the others, especially B, to spew outrageous things, purportedly because he thinks it's funny. And he insults young people all the time, like they are a bunch of losers.
So yeah, it gets toxic when we all get together. I hate to say it, but basically becomes "middle aged straight white guys bitching about everyone and everything that is different, new, etc."
I will report back with how it goes.
PS: I have "divorced" one guy from the group, let's call him J. It happened last year. He had moved away years ago, but was still active on our texting. He has become the most ridiculous Trumper I have ever met. He insults freaking everyone. I flat out said "no more" and left the group. The group created a new text group that didn't include him (but kept the old one too), so basically there are two text groups, one that includes me and one that includes him. But that divorce was easier because he's just not around physically. Also, D followed suit and divorced J as well. He agreed with me that it was too much.