Heh, here in SF some would say a guy wearing his hat backwards means he's a bottom.
Really though, you have to take into account the entire persona, the whole outfit in addition to the circumstances. Are you out at a nice restaurant wearing jeans, tennis shoes and a ball cap backward? Are you in shorts and a tee at the beach? Totally different reaction in my book.