My brother used to smoke weed for years, and he's seriously fucked up. Not necessarily because of the drugs, but it didn't help his situation at all.
Perhaps weirdos are more likely than "normal people" to use drugs, which leads to the general population seeing drug users as having weird problems because of the drugs.