The only reason I can see why you might get offended at the clothing someone wears is if (a) it's got a message written on the clothing that is specifically offensive to you, your gender, your ethnic group, your sexual persuasion, etc. - or (b) the clothing is `special` to your group.
I can't think of any real examples of the last category, but let's suppose you belong to an indigenous group in some country who invented this sacred t-shirt, and only high priests are allowed to wear this t-shirt, and then Tommy Hilfiger comes along and makes a knock-off of the shirt, and you see some guy walking down the street wearing the shirt.
But getting offended or taking issue with a ball cap? If I was the type of person to take issue with a ballcap being worn backwards (or any other clothing item), I'd laugh - and then completely forget about the issue.
