adaviel’s Comments

• Oct 26th, 2010 @ 10:37am

  robots.txt at northcountrygazette.org

  on Local News Website Says You Need To Pay To Read Its Stories, Says It's Collecting Visitor IPs To Sue
  Apparently there's no limit on recursively downloading the site with wget and never reading any warning. Or on reading it via Google's cache (682,000 results):
  
  User-agent: Googlebot
  Disallow: /axs
  Disallow: /cgi-bin/
  
  User-agent: *
  Disallow: /cgi-bin/
  Disallow: /axs/