According To WSJ, Google Not Just A 'Thief' But A 'Digital Vampire'

from the oh-please dept

Thu, Jun 25th 2009 6:02am — Mike Masnick

There's an absolutely huge business out there of folks trying to get more traffic from Google, called Search Engine Optimization. It's a big deal. Traffic to your website is the lifeblood of most internet business models, and so any way to get more traffic is a good thing. Except if you're in the newspaper business for some reason. Lately we keep seeing odd stories of newspaper business folks complaining about the fact that Google sends them traffic. The latest? Dow Jones CEO Les Hinton, who called Google a "digital vampire" claiming that it's "sucking the blood" out of the newspaper industry (found via Mathew Ingram). He then goes on to suggest that at least some of this is the newspapers' own fault for giving "Google's fangs a great place to bite."

So, uh, Mr. Hinton, here's a suggestion: there's a little thing called robots.txt. You can block Google from indexing your websites. Then everyone's happy, right? That stops the bloodflow right there.

Except, perhaps the real issues is that, as everyone in every other business seems to recognize, traffic is important, and it's up to the website receiving that traffic to capitalize on it. So, either Hinton doesn't know this, or he's simply lying. Neither one makes Dow Jones look particularly smart.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: journalism, les hinton, newspapers, search engines, traffic
Companies: dow jones, google

41 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 25 Jun 2009 @ 6:05am

Google doesn't always respect robots.txt
[ link to this | view in chronology ]
- again, 25 Jun 2009 @ 6:09am
  
  Re:
  citation ?
  [ link to this | view in chronology ]
- R. Miles (profile), 25 Jun 2009 @ 6:10am
  
  Re:
  Um, Google doesn't respect improperly written robots.txt files.
  
  One error is all it takes.
  [ link to this | view in chronology ]
  - Anonymous Coward, 25 Jun 2009 @ 6:36am
    
    Re: Re:
    Nope. Google will take links from external sites and try hard to reach those pages even if they are blocked by robots.txt. Google also pushed sitemaps, but then ignores them and uses links from external sites to guide itself to content.
    [ link to this | view in chronology ]
    - Anonymous Coward, 25 Jun 2009 @ 7:00am
      
      Re: Re: Re:
      you are so right. Web developers should not be expected to do their job properly and know how to properly create robots.txt
      
      Also it's not the plumbers fault for not knowing how to soder copper pipes together.
      
      I know it took me an entire 15 minutes to learn how to block all webcrawlers from my online game club's website that had fancy ummm... "sitemaps" and "forums" I guess a professional working at the WSJ shouldn't be expected to do the same amount of work.
      [ link to this | view in chronology ]
    - again, 25 Jun 2009 @ 7:00am
      
      Re: Re: Re:
      citation ?
      
      You can spew all the crap you want, but who is going to believe it without anything to back it up ?
      [ link to this | view in chronology ]
    - Anonymous Coward, 25 Jun 2009 @ 7:02am
      
      Re: Re: Re:
      Well, then you can get in the offensive-
      1. Google uses "Googlebot" as the useragent - block it
      2. Google's IP range is well known - block it
      
      there, I got 2 solutions in under a minute. But that wont stop you from whining... will it?
      [ link to this | view in chronology ]
    - Matt, 25 Jun 2009 @ 7:06am
      
      Security?
      There are abuot 1001 different ways the newspaper could protect its content from spiders.. say store it in SQL database, or link it in from a directory outside of the HTTP server with php, asp, or if they are really worried they can use a C++ DLL and use double password authentication like my online banking (Google doesn't seem to index that)or a Crypto image verification that way only authenticated users can have access to the content. Then when the spider tries to reach that link there isn't any content on those pages because the spider isn't logged in. The problem is they are trying to secure plain html with .htaccess and simple cookie authentication. The problem is with the web designer. Google's spiders are not the only thing out there looking for free content behind paywalls.
      
      Newspapers feel like it is not their job to have to proactively defend their content by writing robots.txt files except every other industry is responsible for protecting its own data online. They are the ones Posting the content, it is their job to protect it.
      [ link to this | view in chronology ]
  - indeciSEAN (profile), 25 Jun 2009 @ 7:13am
    
    Re: Re:
    Google doesn't do the thing they're supposed to do when the person on the other end doesn't write the file correctly? HOW DARE THEY!?
    [ link to this | view in chronology ]
- kilroy, 25 Jun 2009 @ 7:53am
  
  Re: Back that up ...
  A little proof would be nice. Most server software really does work the way it is expected to ... Most of Google's apps work as expected. So show me an instance of Google not respecting a properly written robots.txt
  [ link to this | view in chronology ]
  - Anonymous Coward, 25 Jun 2009 @ 3:07pm
    
    Re: Re: Back that up ...
    A little proof would be nice.
    
    Proof? What proof? We don't need no stinkin' proof!
    [ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 6:14am

what is killing them are their registration and pay walls. If another site has the same news and is able to make a biz model out of not obstructing the news they win and you lose. just part of biz.

Maybe its time for the WSJ to hang it up and go home.
[ link to this | view in chronology ]
- Designerfx (profile), 25 Jun 2009 @ 7:01am
  
  Re:
  not really.
  
  I wouldn't read WSJ even without a pay wall. Lots of people feel that way. They do a bad job as far as newspapers go, they're on the way out as far as relevance.
  [ link to this | view in chronology ]
ed, 25 Jun 2009 @ 6:17am

WSJ Pay Wall
Hmmm,

google is bad, yet content that typically is only available behind their pay wall is purposely exposed for free to google's spider.

Turn that capability off and enjoy your private life behind the pay wall, easy as that.
[ link to this | view in chronology ]
Anshar (profile), 25 Jun 2009 @ 6:44am

Digital Vampire...
That's awesome. Google needs to find a way to own that title. :)
[ link to this | view in chronology ]
- Anonymous Coward, 25 Jun 2009 @ 7:01am
  
  Re: Digital Vampire...
  I vant to suck your binary number systems!
  [ link to this | view in chronology ]
Rob, 25 Jun 2009 @ 6:54am

Do you hear that?
It sounds like the newspaper industry digging their own grave!
[ link to this | view in chronology ]
- Rebel Freek (profile), 25 Jun 2009 @ 9:22am
  
  Re: Do you hear that?
  they must have a hole the size of New York by now...
  [ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 6:58am

The newspapers, of course, are not complaining about being found via Google search. They're complaining about their stories being aggregated into Google's news service, which is different from Google search. But there's no way to opt-out of Google news without opting out of indexing altogether, which would be suicide thanks to Google's near monopoly on web search.
[ link to this | view in chronology ]
- Vincent Clement (profile), 25 Jun 2009 @ 8:58am
  
  Re:
  But there's no way to opt-out of Google news without opting out of indexing altogether
  
  Incorrect. Straight from Google News:
  
  "If you don't want your site to be included in Google News, please let us know and we'll remove it from our index.
  
  Keep in mind that the removal process normally takes a few days and that your articles already included in Google News will expire after 30 days."
  
  Says nothing about being removed from all indexes. Just the index for Google News.
  [ link to this | view in chronology ]
  - Nelson, 25 Jun 2009 @ 6:15pm
    
    Re: Re:
    Ha-ha
    [ link to this | view in chronology ]
- JackSombra (profile), 25 Jun 2009 @ 9:01am
  
  Re:
  You mean the news service that puts about the first 40 words of the article and then links to the source? Yes that is something to complain about...not
  
  Or it is the fact it also links to about a dozen other sources with the same article/subject allowing people to pick who's version they want to read....lot more likely
  [ link to this | view in chronology ]
fogbugzd, 25 Jun 2009 @ 7:27am

Yellow Pages
For a century the yellow pages in the phone book is what drove people to businesses. Search engines have severely damaged that model. Yellow pages are basically a print media, and the though occurred to me why the yellow pages industry hasn't been whining like the newspaper industry.

I think the answer is that the yellow pages industry embraced the Internet fairly early. Now when the yp-type companies call you they promise to get your listing prominently on the web, and then mention that they publish it in the local phone book almost as an afterthought. Newspapers had a choice early on and decided to live in their fantasy world and ignored the Internet. How different would things be if the newspapers had followed the model of the yellow pages industry and realized they do not have a divine right to be protected from change?
[ link to this | view in chronology ]
- Anon, 25 Jun 2009 @ 3:00pm
  
  Re: Yellow Pages
  Except the yellow pages business model has always been about data mining... going on the internet just made their model bigger and more profitable.
  [ link to this | view in chronology ]
JustMe (profile), 25 Jun 2009 @ 7:27am

I think part of the problem
...is that these old guys seem to think there are still only one or two choices for news (occasionally owned by the same company!).

We will still find the same news from another source. The only difference is that you will loose the page views you would have received if you were linked from Google.

Disclaimer -I've had one or two WSJ subs in my life, usually when they were required reading for college classes.
[ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 7:54am

I have a paywall on my site, it's a big page in the front that says "if you haven't paid to read my content, please don't click on the following link". But people and search engine today have no manners!
[ link to this | view in chronology ]
- Anonymous Howard, Cowering, 25 Jun 2009 @ 8:07am
  
  Re: #20 - AC
  And what does the following link say?
  
  "Click to exit"
  [ link to this | view in chronology ]
Yakko Warner, 25 Jun 2009 @ 8:07am

Oh great.
I suppose next we can expect to see a poorly-written teen romance movie about Google now?
[ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 10:12am

WSJ is a News Corporation (Fox, Rupert Murdoch) publication at this point. Their business content may still be first rate, however since it's behind a pay wall I don't know.

From what I've seen of their editorial content, the WSJ has become an echo chamber of capitalist protectionism (now there's an oxymoron). The editorials are against regulation when it would force them to act more ethically, not that I'm sure there is a business - client ethics model at this point. They are for regulation when it protects their business models, which means they do not have to innovate to compete.

It's amazing to me that all these high paid executives cannot figure out new business models to remain profitable. I guess executives earning 300 times the compensation of an average employee is no guarantee of excellence like those executives say it is.

There are models tying together print and electronic media that have the opportunity to work. I've seen some starts in this direction. I've seen some music industry business models that might also be applicable.

I wonder when these executives will start earning their compensation. Maybe this is a question stockholders should ask.
[ link to this | view in chronology ]
Overcast (profile), 25 Jun 2009 @ 10:41am

You know... I suppose those who made stone tablets and chisels would have said the same about newspaper.. Silly new-fangled stuff - don't even have to chisel to put words on it!
[ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 11:06am

Xenu™ Awaits
WSJ should just use the methodology Scientology® Uses.
[ link to this | view in chronology ]
Kyzzyxx, 25 Jun 2009 @ 1:09pm

What is killing the newspaper industry
What is killing the newspaper and TV news industries is your biased reporting and spin of the news. Quit blaming others for what is YOUR own fault.
[ link to this | view in chronology ]
Anonymous Coward, 25 Jun 2009 @ 1:23pm

IT takes money to make money
That's right, when most companies feel technologically outgunned they hire someone to solve those problems for them. This is why people who can operate a "server" should be paid adequately for their skills. So many years have elapsed and did they ever complain about yahoo! news? The newspaper industry should have become google (and wouldnt that be strange?).
[ link to this | view in chronology ]
Mark M, 25 Jun 2009 @ 1:23pm

Other Profit Centers
News should be free, simple. They (Dow Jones/Wall Street Journal) should start optimizing their content better then they already are. Mr. CEO should hit up their advertisers and increase the pricing. I mean, I don't know much about this (its just a thought) but don't major magazines survive off of advertising revenue? I highly doubt Maxim is relying solely on publishing sales to survive. Why do they have to double dip? Maybe Dow Jones should stop releasing content online, and then someone could focus on getting some scholarly content syndicated. Maybe allow some critical thinking to start circulating through people’s blood. For a company that has been around for over 100 years, and is the largest financial news publishers in the world, maybe they should stop their bitching. I doubt Google is the reason for their problems. Blame it on the economy if you want to be that ignorant. I mean people go as far as to blame their weight on it.
-Bitter Google Fan
[ link to this | view in chronology ]
Scott Cleland, 25 Jun 2009 @ 1:46pm

Mike is search advertising competitive like you imply?
Mike
With all due respect, the crux of your strong defense of Google depends on whether the search advertising market is competitive or not. I analyzed Google's slogan -- competition is but "a click away" -- and found it can't withstand close scutiny of the facts or logic. It is untrue and deceptive. See: http://www.precursorblog.com/content/what-one-click-away
Are you officially claiming the search advertisiing business is fully competitive?
Scott Cleland Precursor LLC
[ link to this | view in chronology ]
- Confused, 25 Jun 2009 @ 2:34pm
  
  Re: Mike is search advertising competitive like you imply?
  ---With all due respect, the crux of your strong defense of Google depends on whether the search advertising market is competitive or not. I analyzed Google's slogan -- competition is but "a click away" -- and found it can't withstand close scutiny of the facts or logic. It is untrue and deceptive. See: http://www.precursorblog.com/content/what-one-click-away
  Are you officially claiming the search advertisiing business is fully competitive?----
  
  Ummmm...are you high? Or did you just not read the story you were posting on? I tried applying some close 'scutiny' to the facts and logic of your post, and still fail to see what the competitiveness of the search advertising market has to do with Mike saying that if the newspapers wanted NOT to be listed in Google News, they DON'T HAVE TO BE, but it would be/is completely asinine to remove the traffic they get from Google. I know I shouldn't feed the trolls, but you just challenged him on something that was not said, and insinuated that Google's slogan about competition (which once again has nothing to do with the article) was something Mike is defending.
  
  I just had to know...are you just high, are you a (very weak) troll, or are you just another shill? I can't quite tell, so I figured I'd give you another opportunity to obfuscate the topic.
  [ link to this | view in chronology ]
- Mike Masnick (profile), 25 Jun 2009 @ 3:08pm
  
  Re: Mike is search advertising competitive like you imply?
  With all due respect, the crux of your strong defense of Google depends on whether the search advertising market is competitive or not.
  
  What?!? Dude, I know AT&T pays you a ton to bash Google at every turn, but please, at least keep this relevant.
  
  We're not talking about *search advertising* at all. We're talking about organic search.
  [ link to this | view in chronology ]
Diginess, 25 Jun 2009 @ 3:48pm

Everyone knows that when the WSJ was bought out it ceased to be the WSJ that we all knew. Now it's got a stupid CEO running it, and quite obviously they're on the rocks.
[ link to this | view in chronology ]
Michaelc, 25 Jun 2009 @ 9:34pm

See how they do without google
Google should remove all reference to WSJ for one day from their search results. I am pretty sure WSJ would change their tune as their traffic plummeted.
[ link to this | view in chronology ]
John Davis, 27 Jun 2009 @ 5:00pm

Wow
You have to admit he does raise some good points!

RT
www.complete-privacy.tk
[ link to this | view in chronology ]
Rudy, 28 Apr 2013 @ 10:31pm

Google Stolen My Money!
Google Adsense stolen my money around $400. They suddenly banned my website. Stay away from Google, they are High Risk investing! You will never know when your blog get banned.
[ link to this | view in chronology ]