It has just as much to do with the Drug in many cases as it does the personality of that who takes it.
As we see, it effects different people in different ways. Some get lathargic and lack ambition and motivation, some people gives them help to focus, some people just want to laugh and watch funny movies. However, the mental addiction to the high is there, it's not a physical one no but you get attached to how you feel when you're high and so it effects your relationships with everyone around you.
I've known a lot of smokers, trust me. I've been a musician for a long time and the stuff is so available and so common that I think that at least %80 of the people that I know right now smoke weed on a fairly regular basis (once a week I say is regular) and a smaller percentage of that are very regular up to daily smokers. Some people I know have the ability to handle it and continue to function and others... not a chance. And their relationships suffer.
I feel for the OP, but without a doubt you did the right thing. Since I myself only partake when I am in a certain mood and with certain people I never smoke weed on my own or just for the hell of it, but then you end up in a situation walking into someone's apartment and he/she is by themselves with a bong on their lap, to me that's sad