Trust is everything in a relationship. If you knew of his past, you were willing to give him your trust. Do you trust him?
I agree wiht HamiC, though, it almost sounds as though maybe you're actually looking for something else but afraid to confront that?
As everyone's indicated, communicate with him.
