It's definitely true that you only look at one eye at a time, even though peripherally you see both. Never thought about that.
I think you're over thinking it and of course, if you do concentrate on the "one eye" idea, it can freak you out, hell if I think about it I find it a bit freaky. Makes me want to laugh heh
I think the only way you're going to get over that freaky feeling is if you forget you ever thought of it. Sure it'll pop up again once in a while, but just count to 10 and it will go away again.