Whether the question was appropriate or not, it points toward some insecurity - are they being excluded because of divorce?
We have several divorced friends. When we get caught in a between like this, my usual response is "I didn't notice if X was there, or "I saw X there, but didn't really pay much attention to them." I then follow up with some sort of affirmation - "I'm glad we are still friends, because you are important to me". Then change subject to a question about them: "are you still playing rugby?".
Depends on our level of friendship, of course.