I don't have anything to add, I follow the general consensus that it's probably bothering you more than him.
All I really wanted to say is that the saying is "Ignoring the 500 pound gorilla in the room."
(after re-reading last post) I'm a bit disappointed that everybody is using your example as a specific, because the concept that you're going for is interesting. Personally the only times that I "avoid the gorilla" is if either the response is going to be hurtful to the other person and won't further the discussion or resolve anything, or if it's an issue that's already been discussed repeatedly and we've come to the conclusion that there is no solution or compromise. I had a roommate that was EXCEPTIONAL at bringing up these conversations and she would rail on me to no end that I refused to talk about them, she would push until I would finally snap and say the hurtful thing that I'd been keeping to myself and then she'd get upset at me for being mean.
So my take is highly subjective, I don't do it because it's not worth furthering the fight or because I care about the person that the topic would hurt.