Philangicality

I'm a virgin of 21 years. I am waiting for the right girl to come along but I seem to be having this reoccuring problem with women. I'll see a girl that I'm attracted to and most of the time I'll approach her. I'll talk to her, ask bashic questions pertaining to the situation and sometimes I'll go a little further and start flirting making inuendos, talking about feelings etc. I try to take things slowly. However there are certain girls that I'm a little more aggressive after. I always have sexual thoughts going through my mind with all females but some I tend to get more physical with like grabbing, brushing up against them, etc. I can't really tell if they like it or not since sometimes I get a positive response with a smile and sometimes It'll be a wierd look or whatever. I just can't get the question out of my mind if they enjoyed the experience or not and if I should pursue it further. Any thoughts or suggestions?