The largest sexual turn-on for any of us is ideas. Anime represents the condensation of ideas into their simplest and most direct form.
It is porn that contains only fiction - if any pornography is exploitative to women (a huge debate I can't fathom the right answer to) then anime is not exploitative to anyone, because it's drawings.
In a sense it's porn without the moral grey area of whether the models wanted to do it, or were forced into it by poverty, drugs, violence or whatever.
My own initial sexual awakening came from reading the stories and letters in my dad's (borrowed) magazines - the pictures were nice to look at, but it was the descriptions of how people felt that got me off.
Even now, I love to look at my lover, but I love to hear her excitement more than anything else.
If it dominates your whole sexual life and causes problems you might want to back off a bit - but otherwise, who're you hurting? Lap it up.
If you really want a girl like in the anime, watchout though - you might get one that fucks you and then runs riot with ninja weapons to destroy aliens.