Hmmm... I wouldn't exactly say I'm even at the level of "bi-curious" but I do have an interest in women at some level (whatever it is).
First of all, you're not a freak. Honestly, it's very acceptable for you to be interested in other women. No one's going to look at you differently. However, perhaps you should talk to him about this "longing" of yours and see what he thinks. Hell, he might even be delighted that you're interested in other women.