True, and that's a good point, but whereas the physical appearance of a girl is a limitless array of variables that make it "beautiful" or not, men just have two things: length and diameter.
You're either 10 inches long, or you're not, and all you need for that is a ruler.
Good observation, but I think since the male organ can have a quantitive comparison, where the females have a more general qualitative comparison, the men are simply subject to their own testosterone-driven "bigger is better" mentality.
EDIT: Of course, that's leaving out the fact that whether or not a guy wants it bigger, a prospective lover might have a certain size in mind, and bigger may not always fit the bill.