Google Blocking Other Search Engine Spiders
from the how-incredibly-not-interesting dept
I'm wondering if this is going to be made into a bigger deal than it really is. Google has put up a robots.txt file (just like many many many other sites out there) telling other search engine spiders to get lost. The article here makes it out as if this is a big deal, which I don't think it is. Google just says they're trying to protect their server resources from being wasted on spiders. Some think that Google is doing it to protect some of their intellectual property from getting harvested. The article gets more interesting when it talks about what other sites have on their robots.txt file (if you don't know, if you put a robots.txt file on your site, you can tell search engine spiders which directories not to look in - of course, this also tells less-than-honest people which directories they probably should look in to find the interesting stuff). Anyway, it turns out that eBay's robots.txt file begins with "Go Away" and CNN's says "Robots, scram".
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
