nah, I think its fine.
I've tried to get many a past boyfriend of mine to sit down and watch it with me. Because I love it and it can be very....educational.
I can't blame you for liking it. It's a GREAT show, and it's funny as hell too, always good for some laughs! So, like I said, don't worry about it. Your don't need an excuse to watch it. I know some guys who watch it too.
Fuck the curtains, and your neighbors!