Diet is almost certianally responsible for the drop of testosterone in males. I know because I'm on the testosterone diet. Mine has never been more regular.
I do think that hormones in cattle, poultry, and other common meats and even dairy could be responsible for the ever decreasing age of children entering puberty. I think it represents a dangerous prescedent, and needs to be VERY carefully studied as quickly as possible, as the effects could be detrimental to our reproductive processes.
Or it could just be that we're evolving in order to reproduce earlier.