News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful
from the deep-misunderstandings dept
Following on the speech given earlier this month by the head of the Associated Press, where it was made clear that the AP and news organizations still think that they can be gatekeepers of news, a bunch of publishers along with the AP are now trying to revise robots.txt so that they can hide content on a more selective level. Now, it is true that robots.txt can be rather broad in its sweep. But it's rather telling that it's the publishers who banded together and are telling search engines what changes are needed, rather than working with the search engines to come up with a reasonable solution. In the meantime, there really are some simple solutions if you don't want content indexed by search engines -- but we've yet to fully understand why publishers are so upset that Google, Yahoo and others are sending them so much traffic in the first place.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: publishers, robots.txt
Companies: associated press
Reader Comments
Subscribe: RSS
View by: Time | Thread
Simple solutions:
B: Use a login script.
[ link to this | view in thread ]
I've said it before...
Unfortunately for the publishers no more listings equals no more traffic. Bye-bye AP, bye-bye archaic newspapers that can't make the move into the 21st century.
If the search engines would do this the news publishers would quickly see the value of the listings.
[ link to this | view in thread ]
More info on proposed changes ?
update:
A pdf document with the proposed changes.
[ link to this | view in thread ]
Why it does make sense
You may believe that no such model exists, but bear in mind that a failed experiment is as important as a successful one. For example, if the music industry started off selling unprotected MP3, they would probably get scared of the inevitable piracy, however minor, that would ensue and fold out of the digital media market. Now that they tried various DRM schemes - and failed miserably - they are much more committed to selling unprotected digital media and much more pragmatic as to what is an acceptable level of piracy. Similarly, while such an experimentation period may not help the publishing industry actually find a better Internet strategy, it will certainly make sure they are more informed and committed when they do pick their strategy.
And anyway, having more control can never be bad. Since the courts seem to agree the publishers do have a say as to how their content is used, they might as well use that power to put in place the technology to enforce control. Then they can decide to just continue with the status quo, and change it when they find it beneficial.
[ link to this | view in thread ]
[ link to this | view in thread ]