Twitter Wishes 4.5 Million Osama Bin Laden-Related Tweets Into Their API Cornfield

from the tweets-or-it-didn't-happen dept

Fri, Jun 17th 2011 4:08pm — Tim Cushing

Considering Twitter was instrumental in breaking the story of Osama Bin Laden's death, it seems somewhat strange that they would also be instrumental in limiting access to one of the biggest stories of 2011, if not the decade. (Of course, we're barely into this decade so we probably shouldn't be building these "best of" lists quite yet...) At the center of this unfortunate situation is a dataset constructed from public tweets using either "osama" or "bin laden," which was compiled using Twitter's own API.

Shortly after hearing of Bin Laden's unexpected mortal coil shuffling, Rob Domanski, who blogs as The Nerfherder, was informed of an archive of Osama Bin Laden-related tweets, all packaged up in handy XML format for use with DiscoverText software:

The datafiles were samples taken from live feed Twitter imports starting shortly after the announcement that Osama bin Laden’s death.
Twitter searches for "bin laden" (647,585 documents, 505 MB)
Twitter searches for "osama" (586,665 documents, 451 MB)
This was all for research purposes, however Twitter quickly shut down the project citing their Terms of Service (TOS) Agreement.

Stuart Shulman of DiscoverText had compiled the documents "using an authorized connection to Twitter via their API" which is apparently a violation of Twitter's API Terms of Service. He received an email from Twitter asking him to remove the datasets:

I'm writing about Twitter data being offered for sale on DiscoverText. Scraping the Twitter service is prohibited by our site Terms of Service, and furthermore, resyndicating data obtained through the Twitter API is prohibited by section I.4.a of our API Terms of Service (http://dev.twitter.com/pages/api_terms).

As such, we request you remove the datasets listed at http://discovertext.com/osamabinladen.aspx and any other datasets containing Tweets offered on your site.

Shulman responded:

Let’s be clear. We have never sold a Tweet. The data collected through the Twitter API and shared through our system is the same publicly available data other users capture with screenshots and share on blogs, Facebook or Twitter itself. Nonetheless, the datasets we have assembled and similar samples are being taken temporarily off the Web site pending a resolution of this issue with Twitter.

Well, "temporarily" has turned into "indefinitely." As of June 1st, Shulman's dataset contained 4.5 million Osama Bin Laden-related tweets, all of which can only be marveled at as a REALLY BIG NUMBER but not shared in any usable fashion thanks to Twitter's complaint.

If it's just a "policy first" decision on Twitter's part, it seems a little short-sighted. This information was (and is) of great interest to people worldwide. Perhaps some sort of warning could have been issued instead of a full takedown, thus allowing Twitter to assert its position on API usage without locking up the dataset. Once the dataset already exists, why block it? It's disheartening to see something with as much potential as Shulman's project getting thrown under the TOS bus.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: api, data, information
Companies: twitter

11 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 17 Jun 2011 @ 4:20pm

Wait...
I thought only the evil 'MAFIAA' cared about protecting their content in a digital age? And all these upcoming tech companies were about 'openness' and a new way of doing things?

You people are ridiculous.
[ link to this | view in thread ]
Anonymous Coward, 17 Jun 2011 @ 4:21pm

This information was (and is) of great interest to people worldwide.

I lol'ed. At least you didn't say great importance.
[ link to this | view in thread ]
Anonymous Howard, Cowering, 17 Jun 2011 @ 4:55pm

Maybe...
I wonder if the FBI or some other governmental/cryptically-initialled agency leaned on Twitter...

Time to adjust the tin-foil hat. They can hear you, you know.
[ link to this | view in thread ]
Anonymous Coward, 17 Jun 2011 @ 5:18pm

List of Idiots
This is a fine example of Party A objecting to Party B making A's data more valuable, for free. Mike has taught us well to recognize the pattern. Then A engages in some sort of sleaze to make the data less valuable again. Isn't that the kind of management boneheadedness which is supposed to be restricted to members of the XXAA and the government? Looks like Twitter wants to be added to the list of idiots.
[ link to this | view in thread ]
Rekrul, 17 Jun 2011 @ 5:24pm

"It's a good thing you done there Twitter! A real good thing!" :)
[ link to this | view in thread ]
Anonymous Coward, 17 Jun 2011 @ 5:24pm

How come it wasn't just upped as an xml file over bittorrent?
[ link to this | view in thread ]
Miff (profile), 17 Jun 2011 @ 5:52pm

I'm honestly surprised Twitter still has an API capable of doing anything useful.
[ link to this | view in thread ]
Anonymous Coward, 17 Jun 2011 @ 7:44pm

War on Terror is a hoax...
[ link to this | view in thread ]
Androgynous Cowherd, 18 Jun 2011 @ 1:51pm

I'd like to know under what legal theory Twitter has any right to demand that data taken down. Surely the copyrights in the individual tweets are vested in the individual tweeters, and the United States doesn't recognize a separate database right in a mere compilation of separately copyrighted works where that compilation doesn't add any original creative expression (e.g. in the arrangement) -- likely these archives are just chronological, or even disordered.

So I don't think Twitter has a copyright claim. And I don't see any other IPR being remotely applicable here. Since the tweets were public, trade secrecy clearly cannot apply, and they're obviously not patentable, nor can Twitter have trademarked them, though the tweets may contain trademarks here and there. If there's a publicity rights violation in there anywhere, again the right being violated would be an individual tweeter's and not Twitter's. (That's leaving aside the question of whether a 140-character tweet contains enough creative expression to even be copyrightable at all.)

Which means that Twitter hasn't a legal leg to stand on if that site operator puts the archives back up and keeps them that way. The most Twitter can do about an alleged TOS violation is a) terminate the alleged violator's Twitter account and possibly b) sue for breach of contract. But they have no proprietary interest in that data, legally speaking.
[ link to this | view in thread ]
Tom Landry (profile), 18 Jun 2011 @ 1:58pm

Utterly fantastic title for this piece Tim....lolol
[ link to this | view in thread ]
Josh in CharlotteNC (profile), 20 Jun 2011 @ 6:13am

Re:
I'd like to know under what legal theory Twitter has any right to demand that data taken down.

The data was gathered using Twitter's API.
In order to use the API, you must agree to a TOS.
TOS and EULAs are considered contracts, even though no one reads them.

I'm not a lawyer, but as far as I remember, no one's really been willing to decisively challenge or defend click-through and shrink-wrap TOS and EULAs for fear of a judge making a ruling that turns out to be a precedent.
[ link to this | view in thread ]