Another option might be to attend but not join in the meal. This may be unusual depending on the restaurant. Also, shifts the discomfort to the "host."
This is assuming that your brother isn't buying of course.
Maybe have a beverage, but eat prior. Then you can participate in the celebration, but not in the expensive and non-preferred diet. It's similar to eating out with friends but not ordering a beverage besides water, which is how the bill really adds up sometimes.
This also helps you avoid the biggest pitfall of buffets, which is overeating.