I talked to an old friend of mine tonight, and now I think I'm ready to rescind what I said earlier about psychology being a 'soft' science. I think the lack of mathematical rigor in psychology stems from the fact that psychology is such a new science and also because it's tough to remove all the 'noise' from the system without being sued by said system.
She seemed to think that given enough time, psychology will start to show the same kind of maturity as physics.
A good psychologist follows all the same general guidelines for experiment design and data analysis as any scientist doing 'hard' science. From talking to my friend, it sounds like she knows more statistics than I do, and I have studied a lot of math in my days.
So there you have it: psychology is a science.