Academic Study Says Open Source Has Peaked: But Why?

from the share,-and-share-alike dept

Open source runs the world. That's for supercomputers, where Linux powers all of the top 500 machines in the world, for smartphones, where Android has a global market share of around 75%, and for everything in between, as Wired points out:

When you stream the latest Netflix show, you fire up servers on Amazon Web Services, most of which run on Linux. When an F-16 fighter takes off, three Kubernetes clusters run to keep the jet's software running. When you visit a website, any website, chances are it's run on Node.js. These foundational technologies -- Linux, Kubernetes, Node.js -- and many others that silently permeate our lives have one thing in common: open source.

Ubiquity can engender complacency: because open source is indispensable for modern digital life, the assumption is that it will always be there, always supported, always developed. That makes new research looking at what the longer-term trends are in open source development welcome. It builds on work carried out by three earlier studies, in 2003, 2007 and 2007, but using a much larger data set:

This study replicates the measurements of project-specific quantities suggested by the three prior studies (lines of code, lifecycle state), but also reproduce the measurements by new measurands (contributors, commits) on an enlarged and updated data set of 180 million commits contributed to more than 224,000 open source projects in the last 25 years. In this way, we evaluate existing growth models and help to mature the knowledge of open source by addressing both internal and external validity.

The new work uses data from Open Hub, which enables the researchers to collect commit information across different source code hosts like GitHub, Gitlab, BitBucket, and SourceForge. Some impressive figures emerge. For example, at the end of 2018, open source projects contained 17,586,490,655 lines of code, made up of 14,588,351,457 lines of source code and 2,998,139,198 lines of comments. In the last 25 years, 224,342 open source projects received 180,937,525 commits in total. Not bad for what began as a ragtag bunch of coders sharing stuff for fun. But there are also some more troubling results. The researchers found that most open source projects are inactive, and that most inactive projects never receive a contribution again.

Looking at the longer-term trends, an initial, transient exponential growth was found until 2009 for commits and contributors, until 2011 for the number of available projects, and until 2013 for available lines of code. Thereafter, all those metrics reached a plateau, or declined. In one sense, that's hardly a surprise. In the real world, exponential growth has to stop at some point. The real question is whether open source has peaked purely because it has reached its natural limits, or whether they are other problems that could have been avoided.

For example, a widespread concern in the open source community is that companies may have deployed free code in their products with great enthusiasm, but they have worried less about giving back and supporting all the people who write it. Such an approach may work in the short term, but ultimately destroys the software commons they depend on. That's just as foolish as over-exploiting the environmental commons with no thought for long-term sustainability. As the Wired article mentioned above points out, it's not just bad for companies and the digital ecosystem, it's bad for the US too. In the context of the current trade war with China, "the core values of open source -- transparency, openness, and collaboration -- play to America's strengths". The new research might be an indication that the open source community, which has selflessly given so much for decades, is showing signs of altruism fatigue. Now would be a good time for companies to start giving back by supporting open source projects to a much greater degree than they have so far.

Follow me @glynmoody on Twitter, Diaspora, or Mastodon.

Filed Under: open source, software development

Australian Government Passes Law Forcing Tech Companies To Break Encryption

from the nice-one,-idiots dept

The Australian Parliament has passed a law ordaining compelled access to encrypted devices and communications. The legislation was floated months ago and opened up for comment, but it appears the Australian government has ignored the numerous complaints that such a law would violate civil liberties and otherwise be an all-around bad idea. But that's OK. It's completely justified, according to the Prime Minister.

Scott Morrison, Australia’s prime minister, told local radio on Thursday that encryption laws were necessary to target Islamist terrorism, paedophile networks and organised crime. “These laws are used to catch the scum that try to bring our country down and we can’t give them a leave pass,” he said.

Sure, and if innocent people find their communications compromised by government-mandated holes, so be it. The law was rushed through Parliament in a late evening session since every moment wasted was just one more leave pass for scum. Legislators promise to review the law in 18 months to ensure it hasn't been abused or created more problems than it's solved, but let's be honest here: how often does legislation like this get clawed back after a periodic review? It's never happened in the history of the laws governing our surveillance programs, even after leaked docs exposed unconstitutional practices and widespread abuse of surveillance authorities.

Here's a short summary of the new powers the legislation hands over to law enforcement and national security agencies:

The law enables Australia’s attorney-general to order the likes of Apple, Facebook, and Whatsapp to build capability, such as software code, which enables police to access a particular device or service.

Companies may also have to provide the design specifications of their technology to police, facilitate access to a device or service, help authorities develop their own capabilities and conceal the fact that an agency has undertaken a covert operation.

This law will go into effect before the end of the year. How it will go into effect is anyone's guess. The law provides for compelled access -- including the creation of new code -- but no one seems to have any idea what this will look like in practice. The new backdoors-in-everything-but-name will be put in place by developers/manufacturers at the drop of a court order, with the onus on the smart people in the tech business to iron out all of the problems.

The law only prevents the government from demanding that "systemic weaknesses" be built into devices or programs. Everything else is left to the imagination, including the actual process of introducing code changes in multi-user platforms or targeted devices.

An actual software developer, Alfie John, has put together a splendid Twitter thread pointing out the flaws in the government's assumptions about software development. Since the compelled participants are forbidden from discussing surveillance court orders with anyone (which would include coworkers, supervisors, the general public, etc.), these requested alterations would have to be implemented in secret. The problem is coding changes go through a number of hands before they go live. Either everyone involved would need to be sworn to secrecy (which also means being threatened with jail time) or the process falls apart. Changes ordered by a court could be rejected by those higher up on the chain. Worse, the planned encryption hole could see the compelled coder being viewed as a data thief or foreign operative or whatever.

Law enforcement is going to have to make everyone involved in the product/device complicit and covered under the same prison threat for this to work. The more people its exposed to, the higher the chance of leakage. And if the code will break other code -- or the request simply can't be met due to any number of concerns -- the government make ask the court to hold the company and its personnel in contempt for their failure to achieve the impossible.

To make matters worse, the company targeted with a compelled access request may be monitored for leaks before and after the request is submitted, putting employees under surveillance simply because of their profession.

In some cases, the only weakness that can be introduced will be systemic, which will run contrary to the law. How will the government handle this inevitable eventuality? Will it respect the law or will it simply redefine the term to codify its unlawful actions?

Even if all of this somehow works flawlessly, users of devices and communications platforms will be put at risk. Sure, the compelled access might be targeted, but it will teach users to distrust software/firmware updates that may actually keep them safer. The government may even encourage the forging of credentials or security certificates to ensure its compelled exploits reach their targets. And just because these backdoors theoretically only allow one government agent in at a time, that doesn't mean they aren't backdoors. They may be slightly more difficult for malicious actors to exploit, but once the trust is shattered by compelled access, other attack vectors will present themselves.

It's a terrible law justified by the spoken equivalent of a bumper sticker. And it's going to end up doing serious damage -- not just in Australia, but all over the world. Bad legislation spreads like a communicable disease. If one democracy says this is acceptable, other free-world leaders will use its passage as a permission slip for encryption-targeting mandates of their own.

Filed Under: australia, backdoors, compelled access, encryption, moral panics, secrecy, software development, terrorism

Developer Of Bookmaking Software Gets Full Kim Dotcom Treatment For 'Promoting Gambling'

from the ridiculous dept

The government's war on online gambling is really ridiculous in a variety of ways. Initially, it was driven by a weird mix of moralist groups, backed by offline casinos who didn't want the competition -- all leading to a bill that effectively (through convoluted means) banned online gambling as a part of a law to protect our ports. The casinos are now regretting their initial support for this law, because they want to get into the online gambling world themselves. But the law is in place, and US (both federal and state) law enforcement has supported it and related laws with ridiculous fervor; arresting execs of off-shore gambling sites and even execs of financial institutions that provide payment systems for online gambling sites. As with the over-aggressive enforcement of copyright law, the Justice Department has also seized many websites, sometimes even seizing the wrong websites.

However, this latest story shows just how ridiculous some of these efforts are, and just how far they can go in creating chilling effects. It's the story of the arrest and criminal charges against Robert Stuart, his wife and his brother-in-law for the work they did for their company, Extension Software. The company provides infrastructure to companies that do sports bookmaking, but they were careful to only license the software to companies in countries where such things are legal.

For their troubles they got "the full Kim Dotcom treatment" according to a writeup in Wired:

The case began in February 2011, when Stuart says he and his wife got the Kim Dotcom treatment after about 30 local Arizona law enforcement agents wearing SWAT gear and camouflage dress — some of them with bushes attached to their shoulders to blend into the woods around his house — descended on his home and threatened to send him and his wife to prison for 35 years if he didn’t cooperate.

The search warrant used in the raid said Stuart and his wife were engaged in money laundering, operating an illegal enterprise and engaging in the promotion of gambling. Stuart has tried to obtain a copy of the affidavit used to get the search warrant, but it’s currently sealed.

Yup. They sent in a SWAT team to deal with... a software developer. Because some companies who use his software might possibly have used it to allow gambling in New York (the state that's behind the charges against him). And the warrant itself is sealed, so he doesn't even know why. Then the story gets even worse. In typical law enforcement fashion, they pressured him into doing a plea deal, in which he would have been forced to install backdoor code into his software, and then hack into his own customers to retrieve information on their users any time law enforcement wanted it. He initially agreed to the plea bargain based on bad advice from what he calls "rent-a-lawyers" he found on the internet. However, before it could go before a judge, he wisely backed out.

The NY's DA has defended this effort claiming that it was all perfectly aboveboard and legal to send a SWAT team to threaten a software developer into hacking into his own customers' data for the sake of law enforcement:

Although Daniel R. Alonso, chief assistant in the Manhattan District Attorney’s Office, was reluctant to discuss the terms of the confidential plea agreement or how his office planned to implement them, he insisted there was nothing unlawful about what was proposed.

“The provision you have questioned is perfectly consistent with the obligations of all law enforcement officials to follow state and federal law to secure evidence of criminal conduct,” Alonso said in an e-mail statement. “The staff of the Manhattan District Attorney’s office involved in this case, both prosecutors and investigators, have behaved ethically and consistent with their obligation to seek justice in every case.”

Eighteen months went by... and then last fall he was criminally indicted -- though all of the original charges used in his initial threat had disappeared (because they were bogus) and all the NY DA could come up with was a single charge of "promoting gambling in the first degree." Yes, because he supplies software in places where it's completely legal.

If you think secondary liability issues are important (and I do!), this case should be a major concern. The guy here just developed some useful software and sold it where it was legal to do so. That should not lead to a SWAT team descending on your house, or facing criminal charges and possible jail time. Blame the users of the software if they broke the law, but nothing that these guys did involved "promoting gambling" in areas where it was illegal. It was about providing legal infrastructure (not promotional) software. The whole thing is a shame, and shows how law enforcement's aggressiveness in trying to show themselves as being "tough on crime" can have serious implications.

Stuart notes that he's lost a bunch of his customers since the indictment, which may destroy his business entirely. And this can only lead to greater concern for other software developers as well. If they license software for completely legal purposes, but the software is then used in commission of a crime, are they suddenly (1) going to be threatened at gunpoint to hack into their customers' accounts and (2) then charged on trumped up charges as well?

Filed Under: casinos, gambling, hacking, law enforcement, new york, robert stuart, secondary liability, software development, sports bookmaking
Companies: extension software

Culture Clash Could Doom Microhoo!

from the internal-frictions dept

We don't link to anthropologists much on Techdirt, but Grant McCracken recently wrote a really interesting post on the cultural dimension of corporate mergers. McCracken points out that culture matters a lot in a company's success. Every really successful firm succeeds by developing a distinctive corporate culture that boosts productivity by giving employees a shared set of assumptions about how to do business. In many cases, employees become like the proverbial fish in water: the corporate culture is so pervasive that the employee barely notices it. In other cases, employees explicitly debate the meaning of their shared corporate identity: Google employees commonly debate the most "Googly" way to approach a particular problem.

When two firms merge, the existence of a shared corporate culture can no longer be assumed. Differing cultural assumptions among employees are likely to produce new frictions. One company might be more bureaucratic, work longer hours, have different procedures for evaluating new products, or have any number of other differences in the way they do business. Any of these disagreements can be a barrier to the smooth functioning of the merged company. If the two separate firms are to become a single, organic one, management must find a way to convince employees of the acquired firm to embrace the culture of the acquiring firm. If they fail, the company will limp along with simmering disagreements over strategy and tactics. That was a big part of the problem with the AOL Time Warner merger, for example.

These kinds of cultural assumptions are particularly important in the software industry, where success is often driven by the ability to recruit and retain the most talented programmers. One of Google's key advantage over the last few years has been the perception that it's a more cutting-edge place to work than Yahoo! or (especially) Microsoft. Many smart programmers view Microsoft as too big, bureaucratic, and slow-moving to develop really innovative products, and since they want to work on hot new technologies, they often go elsewhere. Acquiring another multi-billion-dollar company is likely to exacerbate this perception, especially if it causes some of Yahoo!'s best programmers to jump ship in the process.

Filed Under: culture, mergers, software development
Companies: microsoft, yahoo