Just off the top of my head I would say that maybe the detergant or fabric softener your clothes are washed in is doing that to you. Your sweat will bring it out of your clothes and it will get into your pores from that. Though if you don't get it when lifting and you sweat just as much then I don't know what it could be.
Allergic to aerobics