Yes, I take "Ma'am" as an insult. I start looking around for the old lady when I hear that. Personally I'd never use Ma'am for anyone under 60.
The first time someone ever called me Ma'am, I was 21 years old (looked about 14) and it was the neighbor's gutter punk teenage runaway daughter asking me for change on the street. I told her I'd give her change if she never called me Ma'am again and wrote it off as a total lack of social understanding.
I wasn't raised to call my elders Ma'am or sir, and we called my parents' friends by whatever their names were, like "Phil," "John" and "Janis." I picked up Ma'am and Sir somewhere along the line I guess, but I only use it for older folks, in order to adapt to terms of address they are probably more comfortable with, or when the situation calls for making a point of showing respect.
Edit - I guess I don't take it as an
intentional insult to be called Ma'am, but still I wonder exactly how old I'm looking today when I hear it, which is rarely by the way.

I guess this is just a regional difference sorta thing. If someone with a southern accent said it, I'd be less likely to take offense because I'd just figure they had a different custom.