Quote:
Originally posted by DeathTöngue
You're supposed to be able to put a stop to it by adding a robots.txt file... But I have heard some people have gotten mixed results, like a short period of much higher crawler activity.
Just my 2 cents. |
Yep, I am aware of that, Rob. We already block crawlers from indexing our images, for instance, to avoid further problems, all with robots.txt. However, it is good that our pages get reindexed and refreshed by the major robots, so that's why I don't block them.
We'll see what happens in the next couple of weeks... And if thinks don't get better, we'll get another server.