You might not find everyone around you enjoys the same "fetish", but I think you'll find there are LOTS and LOTS and LOTS of people that enjoy it. There are tonnes of websites devoted to it, companies that sell just dirty panties, women that sell their undies through the mail, vending machines in Japan that dispense them...
I think you should revel in it and enjoy it. Imagine, there are tonnes of people out there having plain old boring sex, believing it never really gets any better.. and here you are, getting rock hard turned on and having a blast with a pair of undies
As for your woman, you are both going to have to talk about this if it's an important part of your sexuality. You must have your needs fulfilled, or you will eventually start looking elsewhere. I wouldn't tell her that, it would be a bad way to start the conversation. However, it would be good to discuss it with her, let her know how you feel about it, and find out what her fears are.