I think i know how you feel. I have 5 or 6 groups of friends, with whom i act slightly differently. Because of this, i can't do a supermegazord type meet up because i will then not know how to act and ebnd up alienating one group or another, or i'm not sure how they'll get on etc.
Because they all live in different parts of the country, it's usually not a case of popping out to the pub to see them, and i usually have to plan a specific weekend to meet up with a certain group. And there definitely aren't enough weekends in the year.
Bleargh, i've been neglecting some of them for sure, and i wish i didn't, but what can you do?