It is understandable how you feel. It’s one thing to become close friends with co-workers, but to exclude others in work situations is just plain immature, not to mention ineffective in building a positive work environment. Good for you for standing up to them.
On a side note, there is a personality characteristic called Includer. While we all have it to varying degrees, when it is ranked higher in a person’s assessment results, they often feel the same way that you do in these types of situations. Here is the description. Do you relate to it at all?
Includer
“Stretch the circle wider.” This is the philosophy around which you orient your life. You want to include people and make them feel part of the group. In direct contrast to those who are drawn only to exclusive groups, you actively avoid those groups that exclude others. You want to expand the group so that as many people as possible can benefit from its support. You hate the sight of someone on the outside looking in. You want to draw them in so that they can feel the warmth of the group. You are an instinctively accepting person. Regardless of race or sex or nationality or personality or faith, you cast few judgments. Judgments can hurt a person’s feelings. Why do that if you don’t have to? Your accepting nature does not necessarily rest on a belief that each of us is different and that one should respect these differences. Rather, it rests on your conviction that fundamentally we are all the same. We are all equally important. Thus, no one should be ignored. Each of us should be included. It is the least we all deserve. Source