USA Today May Be A Bit More Social, But It Doesn't Want Google To Know What The Riff Raff Have To Say

from the not-really-letting-them-in dept

Wed, Mar 7th 2007 8:52am — Mike Masnick

USA Today has definitely been a lot more innovative than some other newspapers in trying to figure out how to thrive online. While other newspapers have tried to shrink their audience or shrink their relevance by putting up hurdles that take them out of the greater conversation, USA Today has stepped forward with experiments to embrace citizen journalism and embrace social media tools. None of these experiments are guaranteed to succeed, but it does seem like a good sign that the newspaper is trying and seeing what happens. However, it appears that while USA Today has embraced some aspects of breaking down the barrier between the newspapers and the people it serves, they're still pretty scared of having those people associated with USA Today's brand. The new social features are designed such that the information that people put into them is hidden from search engines like Google.

Matt Marshall over at VentureBeat contacted USA Today to find out why, and was basically told that the newspaper is worried that the general public will give its brand a bad name. A company representative told Marshall that the newspaper is worried "about what content is associated with the brand...." That seems silly however. The comments here on Techdirt are wide open and indexed by search engines -- and while there may be a few people who somehow think that a random commenter represents the views of Techdirt, most people recognize that they're the views of someone else, but posted here. The USA Today representative followed up his statement by saying: "We're still a newspaper." But, that's just the thing. They're no longer really a "newspaper," but a business that's delivering news and information in a useful way to a community of people. Devaluing that community by suggesting they're not worthy of being included in Google isn't a particularly friendly "welcome" gesture.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

11 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 7 Mar 2007 @ 9:28am

No problem
I don't see a problem. When you put up a bunch of road blocks, such as captchas, passwords, and such you keep spammers away, but you keep the useful people away, too. When you take away the benefit of spamming without setting up road blocks, the spammers go away and the useful people stay. Most people don't care if their comments are indexed by Google.
[ link to this | view in thread ]
Anon, 7 Mar 2007 @ 9:47am

This is nothing new, there is a reason Google created the rel="nofollow" attribute, and the whole point behind robots.txt. So that site owners can control what is indexed and what isn't.
[ link to this | view in thread ]
Dan, 7 Mar 2007 @ 9:48am

papaer comments
I run a few sites and many times I feel that very large and off topic comments actually hurt search relevance and targeted ads... So perhaps having the main stories index while not having the comments dilute the story content and such isn't a bad idea... Sometimes comments ad a lot to the topic, but many times it doesn't seem to add directly to the content search value... Since search engines make use of keyword ratio and such in the content it make sense to perhaps control what is included with that somewhat.

Obviously entirely community written pieces should be indexed, but comments on other stories might be a bit different.
[ link to this | view in thread ]
Jim, 7 Mar 2007 @ 9:49am

Doesn't really matter
The new design at USA Toady is so bad I doubt many people will bother with it.

Also, I have a real problem with the concept of news as a "social medium." USA Toady will present stories based on popularity and comments. Geesh, the most important news will become "who had plastic surgery" instead of "who's committing corporate fraud." News is not a popularity contest!
[ link to this | view in thread ]
comboman, 7 Mar 2007 @ 10:16am

Better than the Belgians
At least they are not suing Google like the Belgian newspapers. USAToday is making responsible use of 'robots.txt' to limit what gets indexed, and that's their right. They probably have good reasons (like limiting comment spam) but even if they don't, it's their decision to index only the content that they created. Sounds like a good compromise to me.
[ link to this | view in thread ]
dataGuy, 7 Mar 2007 @ 10:16am

Re: No problem
I think you hit the nail on the head (and with the first post). USA Today may be not indexing comments for the wrong reasons but you have described the real benefit.

Most comment spam is really link spam. If everyone knows the comments are not indexed then only the really dumb spammers will bother to pollute the comment threads. Sounds like a clever idea....
[ link to this | view in thread ]
Kilroy, 7 Mar 2007 @ 10:36am

#3 and #6 I agree with. In addition, this site (techdirt) is designed as a conversation with some pretty advanced topics that are written as opinion and invite conversation both negative and positive.

USA Today, however, will continue to do articles that are straight up journalism and will be marketed towards a much much wider audience, some of whose opinion may be, shall we say, very uninformed?

USA Today is merely looking out for the QUALITY and INTEGRITY of their product.

Imagine a scenario:

A homosexual nightclub burns down in New York City. USA Today writes a whole article about it. Some readers post comments on topic and others use it as a platform to post about their hatred for homosexuals, thus spurning an all out flame war. Googlers do a search on some anti-homosexual rhetoric and up pops the USA Today article.

Now, the article, along with USA Today has been associated with homophobia/hatred.

This is an extreme example and chances are it may be moderated to the point that such an incident will never occur, but, at least it points out a very real concern.
[ link to this | view in thread ]
andy, 7 Mar 2007 @ 11:18am

to a lesser extent
piggybacking on kilroy's comment.

usa today's choice to keep comments from being indexed creates a clear line between the legitimate journalists writing the articles and the often less-equipped and more-volatile story commenters. i think keeping them un-indexed is a great way to say, "the journalist's work is of greater worth and value to our paper than the comments related to that work."

as a journalist, i think it shows respect.

also, i don't know how many people care if their comments show up on searches. if you want your voice to be heard by a wider audience, start a web site... don't forum/comment whore expecting to gain notoriety or some level of (perceived) credibility because of your ability to make remarks on someone else's work. that's weak sauce.

if people are making comments on usa today's web site with the (direct or indirect) intent of showing up on searches, that's kind of pathetic. by removing that incentive, you remove approval-hungry, self-aggrandizing commenters. win-effing-win for usa today.
[ link to this | view in thread ]
Jon (that first commenter up there, 7 Mar 2007 @ 11:21am

Re: Kilroy
Agreed. Also, I'm not just thinking about the article being "associated with homophobia/hatred" but also that the spammer can get his hateful rant at the top of Google results just by commenting on a news article. The spammer isn't out to tarnish USA Today as much as he's out to benefit himself and his own ideas. And most non-spammers don't care, so they're not losing any benefit.
[ link to this | view in thread ]
Sean, 7 Mar 2007 @ 1:28pm

Wow can't search comments
/yawn
[ link to this | view in thread ]
bshock, 8 Mar 2007 @ 7:58am

USA Today?
I'm sorry -- you actually read USA Today?

Do you watch Fox News also?

Honestly, if you are so serious about brain death, slitting your wrists is much faster.
[ link to this | view in thread ]