I also have major problems with computer noise. You don't really notice it if you've gotten used to it, but once you really have a silent PC, you wonder how you ever lived with all the high-pitched whining noises, the rattling, and the noise given off by CPU Fans, case fans, etc....
A great starting point is
www.silentpcreview.com, where they have reviews of many components that have been specially designed to be quiet. For example, quiet CPU fans, quiet Power Supplies, some of the stuff there is a bit over the top for just trying to create a quiet computer, but it's an interesting read at the very least.