I agree with what skysooner said about being at peace with who you are.
Confidence and self esteem can be generated in different ways based on the type of person you are.
Some people's self esteem and confidence regenerates without the need for affirmation from others, and yet for many other people (like me, for example), affirmation and acknowledgement is needed for confidence to exist.
That's why some people will reply that you should stop giving a shit about what other people think, and some will tell you otherwise.
As far as personal matters go, you should recognize that the best solution for you is the solution that fits with who you are. If you ask me, you sound like the type of person that needs affirmation from other people to be confident in yourself.
Develop a better understanding of other people and what motivates them to do what they do, and then take up understanding yourself and you'll begin to see how your confidence is affected by what people say and do. Then you can tackle the problem by coming up with some way to change what people do around you, so that you are primed with great confidence all the time.
I've read a modest amount about typology (psychological types) and that's basically how I came to this conclusion for what you should do to solve your confidence problem.
Here's some links to popular and not-so-popular typological systems, from the most apt to least in my opinion:
http://www.socionics.com/
http://www.enneagraminstitute.com/
http://www.geocities.com/ptypes/
http://www.keirsey.com/matrix.html/ (just don't bother with this system. it's the most popular one and yet it has a few fallacies that are fixed in the socionics system)
And if you do follow all this and then want to test to check your own personality type check out this site:
http://www.socionicstypeassistant.com/stc/home.html/
Bottom line:
Some people can care less what other people think, some can't. And if you can't, it probably explains why you have such a problem with it, and you need to be more aware of yourself so that you can spot why you have inconsistencies in your confidence, so that you can make the right changes to fix those inconsistencies.