Primal claims to feel me up regularly while I'm asleep, but I've never really known if he was joking or not.
I'm fairly certain that he wouldn't try to have sex with me though. He'd probably be afraid of what I would do to him when I woke up swinging, confused about what's going on. I don't see anything wrong with it other than I would wake up disoriented and panicked.
I do see why some might call it rape. I wouldn't look at it that way for my particular situation, but if you're not in a really secure relationship or something then it could definitely be looked at that way.
That being said, I think I'll try it the other way around one of these days. I'm sure my husband would love to wake up with me "raping" him.