j, I think you should feel lucky that you got out when you did. He was not being fair and it doesn't sound like he was honoring your feelings. Confidence takes a long time to build, which was mentioned earlier. It also helps if someone is helping you build it, which to me it doesn't sound like he was. If anything, he's hurt your confidence more than build it up. Cut your losses and focus on you. Exercising and doing things that you enjoy are definite confidence boosters. One day you will find a guy who will compliment you and help you shine. But you don't need a guy to do that for you.
And what jj said is true...give it time and build yourself up...one day he will see you and kick himself in the ass for letting you go. (I speak from experience
)