It's much worth the risk. As long as you are smart about it, you'll be fine. I don't go riding during rush hour... I don't put myself in situations in which accidents are more likely to happen. I ride with a helmet, armored jacket and armored pants with ballistic material and hard armor inserts. If I crash, my head will be protected and my skin will not bet ripped off from sliding on the road. The lack of helmet and armor does significantly affect safety... out of all of the local fatal incidents since I've got a bike, I'm pretty sure none of them were as armored as I was and most of them were not wearing helmets.
FYI... to show what a difference armor can make, watch some moto gp racing you tube. those guys have ridiculously dangerous looking crashes during races quite often, they usually are going well over 100mph while ending up sliding and tumbling on the pavement and though the grass, only to smash into walls while their bikes are flying left and right...they almost always stand up and walk away, sometimes picking the bike back up to finish the race if it's not too beat up. A Harley rider wearing a vest and mini-helmet would have died long before hitting anything from the slide alone.
edit:
discovery channel's 'Twist The Throttle" did a good job of talking about armor a while ago... they did a segement on alpine star.