|
Cuil's search robot ignores robots.txt and spiders at an unrestrained pace. The result is that is causes websites to crash and indexes content that its not supposed to. Its brought down a couple of our (my work's) sites so far. We had to block its range of IPs. What a pain in the ass.
|