Do you feel nuts? It's pointless to worry if someone will think you're insane for sleeping with stuffed animals. They will. Trust me.
But then some other people will think I'm insane for NOT sleeping with stuffed animals. As long as your behavior and preferences does not hurt you or others, then what's wrong with it? Enjoy your stuffed animals