That is a small price to pay for your safety. Anything could happen with him dealing drugs out of the next room. If the druggies know where he lives, they could break into the apartment looking for his stash. If these people are in and out of your home you'll probably find things start to go missing. Hell, the cops might catch wind of it and throw your ass in jail with him.
Get out while you still can.