When Your Abnormal Is Not The Normal Abnormal

I was just reading the Want to try swinging thread and I get that this is a site centered more around discussions and not meant for hooking up, but this is a discussion about bringing other people into your relationship, finding suitable people and all the difficulties and frustrations that can go with that.
We are not swingers. If there is a 'typical' abnormal lifestyle, swinging seems to be it. What about couples or groups looking for one other category of person/people to join them? We happen to be a male-female couple looking only for other women (most heterosexual men's dreams, right?) and we find a huge shortage of suitable partners. But I am sure this frustration extends to any couple who wants to involve other people in their intimate lives without full on swinging or swapping. As the woman in this relationship and a voyeur, this is also an important part of my sexual desires that we have trouble fulfilling.
I am not looking for a list of sites, we've tried them. I doubt we can walk into a swingers club and say thanks, but we're only interested in your wife/girlfriend. My BF has slept with other women and we enjoy talking about it, but it always ends the same, they're uncomfortable with the idea of having me around or they get too close to him and think they can get more from the relationship.
We can't possibly be the only ones who experience this???