Australia Sets New Overblocking Record: Aims For One Site, Takes Down 250,000

from the whoops dept

Thu, Jun 6th 2013 11:57am — Glyn Moody

Overblocking is not a new problem -- over two years ago, Techdirt wrote about an instance where Homeland Security took down 84,000 innocent sites at a stroke -- so you might have thought that those employing this blunt instrument would take a little more care these days. However, things seem to be getting worse, not better. In Australia, the Australian Securities & Investments Commission (ASIC) has just scored a whopper:

The largest number of sites censored when attempting to block one particular site ASIC believed was defrauding Australians was 250,000. Of these, ASIC said about 1000, or 0.4 per cent, were active sites. It said the 249,000 other sites hosted "no substantive content" or offered their domain name up for sale, rather than hosting a fully-fledged active site.

I wonder how the ASIC established that 249,000 had "no substantive content". I can't believe it really checked all of them. And that's the big problem with overblocking: when huge numbers of sites get taken down by mistake, there's no way of telling what just fell off the Web, and what obscure but possibly important information is no longer available.

At least ASIC realizes that its current ham-fisted approach isn't acceptable:

ASIC told senate estimates in its opening statement that it was now examining how it could ensure only a site's specific domain name was blocked and ways it could alert the public to a site being blocked via a pop up page. It was also examining ways such a page could indicate why access was blocked and to whom queries could be made to dispute a block.

That's good, but maybe it would have been better if it had explored those options before shutting down nearly a quarter of a million innocuous sites by mistake.

Follow me @glynmoody on Twitter or identi.ca, and on Google+

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: australia, copyright, overblocking, takedowns

50 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 6 Jun 2013 @ 12:18pm

The 249K number reminds me of the number in some ads I saw about how one supermarket claimed 1.5 million customers where happy with them, when all they did was to take a sample of 200K and extrapolate the rest, that ad got banned in the UK for being misleading.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 12:18pm

Overblocking
Preventing access to even a single website is overblocking. The proper way to shut down a website is to go through whatever legal process is due at the location in which it's hosted.
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 12:26pm

Several points
> I wonder how the ASIC established that 249,000 had "no substantive content".

Forget substantive content. It was all just an anomaly right? Or would this be collateral damage? You can expect some collateral damage in a city if you use a 45 megaton nuclear weapon to kill an ant.

Whether content is substantive or not, people have a right to say it and not have it taken down by private interests that cannot be bothered to exercise even the smallest bit of care in their use of nuclear weapons.

From the article . . .
> This meant thousands of other sites were blocked in the process,
> as many sites are often hosted on one shared IP address.

But wait. I thought an IP address was equal to one person, not a quarter million websites!
[ link to this | view in thread ]
ChurchHatesTucker (profile), 6 Jun 2013 @ 12:37pm

Duh
ASIC told senate estimates in its opening statement that it was now examining how it could ensure only a site's specific domain name was blocked

Just, y'know, block the site's specific domain.

Seriously, are Joeys in charge of this? Because that would at least be cute.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 12:38pm

Great job guys!

It's so comforting that people like this who CLEARLY know what they're doing are the ones making the decisions regarding what websites are online and what ones aren't.

/sarcasm
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 12:45pm

Re: Duh
"ASIC told senate estimates in its opening statement that it was now examining how it could ensure only a site's specific domain name was blocked"

...and I'm just spit-balling here...but wouldn't it be prudent to know how to do this BEFORE you start doing it?
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 12:45pm

Re: Duh
> Just, y'know, block the site's specific domain.

I'm sure they will follow your wise advice and they'll block, let's say for example, blogspot.com.

Then we'll hear from the idiot brigade that it's just an anomaly.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

out_of_the_blue, 6 Jun 2013 @ 12:54pm

Well, Techdirt usually has "no substantive content"...
"I can't believe it really checked all of them." -- Never heard of automation, eh? Tip for you: computers can be programmed to download and analyze the text on sites. One corporation, named "Google" does this surely millions of times an hour. Any half-decent programmer could knock out an adequate script in under an hour.

But I admire the chutzpah of assuming on zero direct evidence that you're right and the "ASIC" is wrong. You have grokked the Techdirt way. Gold star for you.

SO, with that out of the way, you're left with a 1000 sites that MAY need to be switched or otherwise adjusted, also by a script.

But in ANY case, the cause is not from ASIC, it's from alleged criminals, which Techdirt always tries to protect.

BUT WAIT A SEC... OMG! The one site out of hundreds of millions with THE ANSWER has been taken down! Humanity is DOOMED!
[ link to this | view in thread ]
Robert Sund (profile), 6 Jun 2013 @ 12:56pm

ASIC
Why the surprise?

ASIC have always been synonymous with single-mindedness and applying that inflexibility quickly and efficiently to large numbers in parallel!
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 12:59pm

Re: Overblocking
What if the legal process involves blocking it?
[ link to this | view in thread ]
Zakida Paul (profile), 6 Jun 2013 @ 1:01pm

Re: Well, Techdirt usually has "no substantive content"...
"Well, Techdirt usually has "no substantive content""

You do realise that your ramblings are not the only thing on this site, right?
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:02pm

Re: Well, Techdirt usually has "no substantive content"...
Any half-decent programmer could knock out an adequate script in under an hour.

To crawl the sites, maybe. But to decide what is "substantive", let alone to decide what qualifies as "substantive" is hardly so trivial.
[ link to this | view in thread ]
Vidiot (profile), 6 Jun 2013 @ 1:02pm

Re: Several points
"I thought an IP address was equal to one person..."

Yes, and Mr. Prenda would like to know who that one person is. He'd like them to send him money.
[ link to this | view in thread ]
ChurchHatesTucker (profile), 6 Jun 2013 @ 1:07pm

Re: Re: Duh
Yeah, we'll probably have to teach the Joeys what a "site specific domain" means.
[ link to this | view in thread ]
Vidiot (profile), 6 Jun 2013 @ 1:09pm

Re: Well, Techdirt usually has "no substantive content"...
"... the cause is not from ASIC, it's from alleged criminals..."

Ahh, our old friend, "alleged". What if it were possible to, say, suspend someone's ISP account... not saying who... for being an "alleged asshat"?

Guess you're right... "alleged" can be useful after all.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:13pm

Pop up page?
What about when the website uses https, with the modern certificate pining stuff? The user would get an unskippable certificate error.

And with SNI, which is already usable for everyone except very old browsers, you can have thousands of https sites on a single IP.

With the IPv4 address crunch, many https sites on a single IP will become more common, and https is becoming more and more popular. So this pop up page idea gets even harder as time passes.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:25pm

This particular task isn't that hard
I wonder how the ASIC established that 249,000 had "no substantive content". I can't believe it really checked all of them.

Why not? If you know how to use curl or wget combined with a little perl or python it's fairly easy to discern which web sites aren't really web site, but single-page placeholders being used by domaineers. All those are cookie-cuttered: they have the same markup, the same layout, (nearly) the same content, (nearly) the same links, etc. The only reason they exist is that these scammers are hoping someone will typo a URL and land there, generating a fraction of a cent's worth of income.

The computing resources needed to do this are minimal: with a decent laptop and a modestly-fast connection, this can be done overnight.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:27pm

Re: Well, Techdirt usually has "no substantive content"...
Who gives a shit about automation? What the hell does THAT have to do with the issue?

Why not focus on the REAL problem - one site needed to come down and they instead took down 250,000!

Great job missing the point AGAIN, retard. Funny - I don't think you'd see it so trivially if it was YOUR site.
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 1:30pm

Re: Re: Overblocking
What if the legal process involved blocking anything within a 50 mile blast radius?
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 1:34pm

Re: Re: Re: Duh
Wouldn't more specific mean further to the right. So, for example, dot-com would be more specific than, say, blogspot-dot-com.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:35pm

Re: Well, Techdirt usually has "no substantive content"...
So what about the 1,000 or so that DID have substantive content?

Tough shit?

And the cause IS from ASIC, moron. They initiated the order that took them down (or did you miss that when you "read" the article?).

I just don't understand your way of "thinking" blue. How exactly can you defend gross incompetence of this magnitude over and over again? Or is 250,000 sites just an anomaly?
[ link to this | view in thread ]
horse with no name, 6 Jun 2013 @ 1:36pm

I wonder how the ASIC established that 249,000 had "no substantive content"
I wonder how the ASIC established that 249,000 had "no substantive content"

Oh, i don't know... maybe by looking at access logs? Maybe by looking at how little traffic was actually coming in?

Actually, in this case it looks like a parking page site, because all 250,000 were apparently on a single IP address or had DNS from the domain in question.

Perhaps rather than just repeating what the original writer put up, you might want to actually put some effort into understanding. Your blind outrage is amusing, by sort of stupid.
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 1:37pm

Re: Re: Well, Techdirt usually has "no substantive content"...
You've got to remember, he's paided by Hollywood, so 'substantive' is defined as bulk volume of number and/or size. A $200 Million dollar movie with no plot is better than a $20 Million dollar movie that wins awards, for example.
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 1:38pm

Re: Re: Well, Techdirt usually has "no substantive content"...
> Great job missing the point AGAIN

He's not missing the point.

He's distracting from the point, which is what he's paid to do, and what he so consistently does.
[ link to this | view in thread ]
Baldaur Regis (profile), 6 Jun 2013 @ 1:39pm

I wonder how the ASIC established that 249,000 had "no substantive content".
Robert Peasley, the ASIC intern assigned to visit each and every one of those 249,000 sites, reported "they contained nothing but cat pictures." Peasley is currently "resting comfortably in a cat-free institution."

But for real humor, there's this:
More than 87% of active domain names are found to share their IP addresses (i.e. their web servers) with one or more additional domains, and more than two third of active domain names share their addresses with fifty or more additional domains.
This quote is from an article abstract. The punchline? The article was written in 2003. How long ago was that in internet years?
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:39pm

Re: Well, Techdirt usually has "no substantive content"...
A two line site saying child protection, phone 12345678910 is substantive. A site that collected all your whining and ad-homs would not be, even if it had 10,000 pages crammed full of text.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:42pm

Re: Re: Well, Techdirt usually has "no substantive content"...
(Hit return too soon).
I should also add that blocking your site, if you had one, would still be objectionable as censorship. You have the right to free speech, even if you add nothing to discussions.
[ link to this | view in thread ]
DannyB (profile), 6 Jun 2013 @ 1:42pm

Re: I wonder how the ASIC established that 249,000 had "no substantive content"
horse with no brain wrote:
> Oh, i don't know... maybe by looking at access logs?
> Maybe by looking at how little traffic was actually coming in?
> Actually, in this case it looks like a parking page site

So you're saying that commercial advertising of domain names for sale, or parking sites for domain names purchased but no web site yet set up means it is okay?

So it's okay to deprive advertisers of revenue? And it's okay to deprive the owner of a domain name the right to begin setting up their new website on their new domain?

Got it.

And I'm sure this was done with the utmost care and with the strongest of evidence. Not just some insane accusation based on an IP address or single domain name.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 1:53pm

Re: Re: Re: Overblocking
They better make it 100 just to be sure.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 2:04pm

Re: Re: Overblocking
Then the legal process needs fixing.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 2:39pm

i read where all of this sort of thing has to be OK'd by Gillard first. it seems as though she's not a lot of good at a lot of things, and sorting out the 'net' is one of them
[ link to this | view in thread ]
angelbar (profile), 6 Jun 2013 @ 2:51pm

Re: Re: Re: Re: Overblocking
-From orbit-
[ link to this | view in thread ]
John Fenderson (profile), 6 Jun 2013 @ 3:17pm

Re: Well, Techdirt usually has "no substantive content"...
Any half-decent programmer could knock out an adequate script in under an hour.

This actually made me laugh out loud.

The best programmer in the world couldn't create a system to do this in an adequate manner in a month, let alone an hour, unless you're talking about downloading a small number of sites and your idea of "analysis" consists of nothing more complicated than scanning for keywords.
[ link to this | view in thread ]
John Fenderson (profile), 6 Jun 2013 @ 3:25pm

Re: This particular task isn't that hard
The problem with that is that you can't tell by the web site.

For example, I have a personal domain name that I use for email and other private services. There is no website attached to it at all -- if you browse to my domain name, you'll get the placeholder page you're talking about.

This domain is heavily used every day and critical to a reasonable number of people. It would have been blocked as having "no substantive content" anyway.

The web != the internet.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 4:42pm

Re: I wonder how the ASIC established that 249,000 had "no substantive content"
So, if 250,000 Music albumns or movies were removed from the major studios when attempting to remove 1, then that would be fine?

250,000 books removed from a publisher when attempting to remove 1, that would be fine?

Collateral damage is fine when it is not your industry on the receiving end?

Wanker
[ link to this | view in thread ]
horse with no name, 6 Jun 2013 @ 6:42pm

Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
Way to miss the point. 250,000 blank websites lost is not a loss, except perhaps if your thrill in life is surfing parking pages and "coming soon" websites.

Is there any true collateral damage here?
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 8:15pm

Re: Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
What was the standard used to determine if there was substantive content on the site?

Who detrmines substantive content? the media industry?

If it is not content from the legacy entertainment industry then it does not count as substantive content?

You would think that it is collateral damage if one of those 250,000 sites was yours.

This is the definition of 'Intellectual Property THEFT', not Intellectual property infringement that copyright maximalists like to conflate with theft
[ link to this | view in thread ]
The Old Man in The Sea, 6 Jun 2013 @ 8:20pm

Re: Several points ASIC not private interests
Just clarifying, ASIC is a government organisation that is supposed to keep businesses honest in Australia. So sites that have a purpose of defrauding the public are within their purview.

However, ASIC doesn't have a perfect record for managing this part of their portfolio. They are not the brightest "Cube" on the block. Though that can be applied to many parts of the Federal Government here in Australia.

As one national newspaper commentator said many years ago (paraphrased). The fundamental thinking in any government department at any level of government in Australia makes the same basic assumption, "all members of the general public are convicts and must be treated as such."

Hence, if taking down one site means that multiple other sites (100's of 1000's) are also affected then it really doesn't matter as they are all problematic as well.
[ link to this | view in thread ]
The Old Man in The Sea, 6 Jun 2013 @ 8:33pm

Re: I wonder how the ASIC established that 249,000 had "no substantive content"
Having worked with these type of people before, I would guess that the whole process was initiated by some non-technical person who didn't bother to get the necessary facts before issuing the order.

In the aftermath of the problem, there would have been an analysis of the pages taken down by personal view (looking at maybe five or six sites). This analysis would then be used to justify what has happened and to colour the view.

Anybody with any technical expertise would very likely not have got their input past along - technical people are just too freaking dangerous to allow their opinions or views to be passed up the chain of reporting. They tend to highlight problems with the various approaches being considered.
[ link to this | view in thread ]
The Old Man in The Sea, 6 Jun 2013 @ 8:46pm

Re: Prime Minister Gillard to you
Please have some respect for the function of Prime Minister. It is a hallowed position in the governance of Australia.

She is a very intelligent woman. She was able to shaft the previous Prime Minister quite easily without having suffered the same consequences herself. She is leading Australia back to the greatness that was called "The Dark Ages" with her various policies for national development.

Security is a high priority for her and her government that they will use whatever means possible to keep all of the people safe by watching their every action.

She follows in the footsteps of those great previous Prime Ministers Robert Hawke and Paul Keating. Men of such renown that the first declared that there would be no children living in poverty in Australia by the year 2000 and the second declared that Australia was a "banana republic".

So please respect her and the position she holds by addressing her correctly as Prime Minister Gillard.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 9:26pm

Re: Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
You know damn well you'd call it collateral damage if it was your site being wrongly taken down.
[ link to this | view in thread ]
horse with no name, 6 Jun 2013 @ 10:50pm

Re: Re: Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
What was the standard used to determine if there was substantive content on the site?

I don't know - maybe they were all parking pages with the same code? maybe they were all single pages? Maybe they were all identical except the name?

Do we know if it was 250,000 domain names, or just 250,000 third level domains of the same site?

I think you might want to cool your jets until you know more, because you are certainly going off the handle with limited working material here.
[ link to this | view in thread ]
Anonymous Coward, 6 Jun 2013 @ 11:06pm

Re: Re: Re: Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
so you were just making up facts when you stated that it is 250,000 blank websites?

Way to miss the point. 250,000 blank websites lost is not a loss

How can you state that it is not a loss if you do not even know what was removed?
[ link to this | view in thread ]
Anonymous Coward, 7 Jun 2013 @ 12:43am

Re: Re: Re: I wonder how the ASIC established that 249,000 had "no substantive content"
See this post about a domain being used for email.
[ link to this | view in thread ]
horse with no name, 7 Jun 2013 @ 2:15am

Re: Re: This particular task isn't that hard
Yes we understand that John. However, what is the chance of all 250,000 domains on a single IP being exactly the same, having absolute no content? Powerball odds come to mind here.

Your argument would only hold water in a single or very small number of domains. 250,000? Better chance that you get hit by lightning while partying with the Yankees - in a subway station.
[ link to this | view in thread ]
Pragmatic, 7 Jun 2013 @ 5:35am

Re: Well, Techdirt usually has "no substantive content"...
If Techdirt really has no substantive content, Blue, why are you here every day, rubbing your ass all over it like a dog with worms?

But in ANY case, the cause is not from ASIC, it's from alleged criminals, which Techdirt always tries to protect.

Read this: http://en.wikipedia.org/wiki/The_Trial

Then this: http://en.wikipedia.org/wiki/Due_process

BUT WAIT A SEC... OMG! The one site out of hundreds of millions with THE ANSWER has been taken down! Humanity is DOOMED!

250,000 ≠ 1. Go back to school, troll.

Wait, if you can't even count to one, how in the world are you going to be able to read those linked articles? Does your mommy type your posts? :S
[ link to this | view in thread ]
Pragmatic, 7 Jun 2013 @ 5:46am

Re:
Shared servers. My website is hosted on one of these. That is not and never will be a good enough excuse to take a website down.

And when you're in the process of building a website, i.e. adding substantive content, it's common to put up a Coming Soon page.

Go Daddy provides a revenue stream for people who buy up domains by letting them use them for advertising. And Go Daddy offers a shared hosting service.

NOW can you see what the problems are, Horse's Ass with no brain?
[ link to this | view in thread ]
John Fenderson (profile), 7 Jun 2013 @ 9:34am

Re: Re: Re: This particular task isn't that hard
I'm not sure how that's relevant.
[ link to this | view in thread ]
Dave, 7 Jun 2013 @ 12:27pm

Re: Well, Techdirt usually has "no substantive content"...
One word about OOTB - laughable. What's it like to be a laughing stock, Blue?
[ link to this | view in thread ]
Anonymous Coward, 10 Nov 2013 @ 11:01pm

horse with no name loathes it when due process is enforced.
[ link to this | view in thread ]