If there was only a way to get the rest of the group to turn on Fugly...then you'd be free to focus your attentions on all the other, less fugly ones
If it were me, I'd prolly go for broke. Let it be known (you don't even have to be subtle about it) that you don't "belong" to anyone and you'll sit wherever you want to and talk to whomever you bloody-well like.
No regrets.