Am I attracted to women from time to time - sure. Not often, but I won't deny it when it's there. But, I don't think that I've ever thought I was a lesbian because of any fantasy I've had about women - I think it just means that I've been curious at different points in my life. I know my feelings on men and I could never give them up.
That said, I agree with Onodrim, as long as you're happy with it, I don't think it matters who you're attracted to.