Panda: a wank every now and again won't be a problem, as far as I know. in fact, it may help you figure out what you like, which you can then turn around and share with your partner, meaning MORE and easier orgasms, instead of the opposite.
also, a general tip: don't give up the vibe altogether, just don't turn it on!
I know i enjoy the feeling of something in me, while I rub my clit, whether it's buzzing or not!