I think my bike cost about $500 altogether. I bought the frame and the drive train components myself and then my awesome roommate bought the parts for and put together a wheel set for me. I like it a lot. It's a fixed gear Surly Steamroller.
Sometimes they shut the freeways down.
I also know people who wouldn't even consider buying a new bike, but since I live in a pretty bike friendly town, there are many cheap used bikes to be had. If you go that route, or actually even if you don't, you'd probably be better off if you also familiarize yourself with the most common maintenance tasks, like changing a flat.
A lot of people who ride fixed gears swear by them. They are about as simple as you can get with a bicycle without going back to the 1820s.