|
if you have a website that doesn't have permissions built into it like a invision or vbulletin, but just a raw HTML site with directories, you can limit spiders with a small file robots.txt. It is a nice convention to tell spiders please don't crawl this directory.
__________________
I don't care if you are black, white, purple, green, Chinese, Japanese, Korean, hippie, cop, bum, admin, user, English, Irish, French, Catholic, Protestant, Jewish, Buddhist, Muslim, indian, cowboy, tall, short, fat, skinny, emo, punk, mod, rocker, straight, gay, lesbian, jock, nerd, geek, Democrat, Republican, Libertarian, Independent, driver, pedestrian, or bicyclist, either you're an asshole or you're not.
|