LEad, it sounds like you have some trust issues when it comes to women that you might not have when it comes to men. And considering the rumor issue, i can see why that is.
I have a couple friends that are women, but both are over 40 and i'm 23, i just get along better with them than gals my own age... so hence, it doesn't matter what age or gender your friends are, just count yourself lucky to have people you can love and trust with your friendship.
So, Personally, i don't think it matters that all your friends are guys, except as you get older, you will find that guys are more difficult to be friends with, since they are usually going to start hitting on you (irregaurdless if you have a man or not)
Sweetpea