Regardless of what the situation is you should not be made to feel like a whore. The past is in the past and just because you've had more then him as a partner doesn't mean you're a bad person, it doesn't mean you're not lovable and it certainly doesn't mean you're not worthy of respect. Talk to him about it, you're his girlfriend, he should love and respect you no matter what. He's probably not aware how hurtful he's being by saying these things. If he stands by his story then maybe you should look for a guy who loves YOU and is not so fixated and insecure about your past.
Good luck.