You went home with him and didn't give him the sex. He figured you weren't interested after he showed interest. He offered you his number so you'd call if you were interested. You refused. You didn't want sex and you didn't want his number, so why would he believe you're actually interested? He's not going to call you.
Or, he's in another relationship already and just wanted some strange this weekend. He's still not going to call because you've already demonstrated that you won't put out. He'll move on.
Or, he's one of those guys who thinks he has to wait a few days to call you for whatever reason.
Those are the three scenarios that come to my mind. My opinion is the first, possibly the third, probably not the second. Good luck.
