Academic Study Says Open Source Has Peaked: But Why?
from the share,-and-share-alike dept
Open source runs the world. That's for supercomputers, where Linux powers all of the top 500 machines in the world, for smartphones, where Android has a global market share of around 75%, and for everything in between, as Wired points out:
When you stream the latest Netflix show, you fire up servers on Amazon Web Services, most of which run on Linux. When an F-16 fighter takes off, three Kubernetes clusters run to keep the jet's software running. When you visit a website, any website, chances are it's run on Node.js. These foundational technologies -- Linux, Kubernetes, Node.js -- and many others that silently permeate our lives have one thing in common: open source.
Ubiquity can engender complacency: because open source is indispensable for modern digital life, the assumption is that it will always be there, always supported, always developed. That makes new research looking at what the longer-term trends are in open source development welcome. It builds on work carried out by three earlier studies, in 2003, 2007 and 2007, but using a much larger data set:
This study replicates the measurements of project-specific quantities suggested by the three prior studies (lines of code, lifecycle state), but also reproduce the measurements by new measurands (contributors, commits) on an enlarged and updated data set of 180 million commits contributed to more than 224,000 open source projects in the last 25 years. In this way, we evaluate existing growth models and help to mature the knowledge of open source by addressing both internal and external validity.
The new work uses data from Open Hub, which enables the researchers to collect commit information across different source code hosts like GitHub, Gitlab, BitBucket, and SourceForge. Some impressive figures emerge. For example, at the end of 2018, open source projects contained 17,586,490,655 lines of code, made up of 14,588,351,457 lines of source code and 2,998,139,198 lines of comments. In the last 25 years, 224,342 open source projects received 180,937,525 commits in total. Not bad for what began as a ragtag bunch of coders sharing stuff for fun. But there are also some more troubling results. The researchers found that most open source projects are inactive, and that most inactive projects never receive a contribution again.
Looking at the longer-term trends, an initial, transient exponential growth was found until 2009 for commits and contributors, until 2011 for the number of available projects, and until 2013 for available lines of code. Thereafter, all those metrics reached a plateau, or declined. In one sense, that's hardly a surprise. In the real world, exponential growth has to stop at some point. The real question is whether open source has peaked purely because it has reached its natural limits, or whether they are other problems that could have been avoided.
For example, a widespread concern in the open source community is that companies may have deployed free code in their products with great enthusiasm, but they have worried less about giving back and supporting all the people who write it. Such an approach may work in the short term, but ultimately destroys the software commons they depend on. That's just as foolish as over-exploiting the environmental commons with no thought for long-term sustainability. As the Wired article mentioned above points out, it's not just bad for companies and the digital ecosystem, it's bad for the US too. In the context of the current trade war with China, "the core values of open source -- transparency, openness, and collaboration -- play to America's strengths". The new research might be an indication that the open source community, which has selflessly given so much for decades, is showing signs of altruism fatigue. Now would be a good time for companies to start giving back by supporting open source projects to a much greater degree than they have so far.
Follow me @glynmoody on Twitter, Diaspora, or Mastodon.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: open source, software development
Reader Comments
Subscribe: RSS
View by: Time | Thread
The free market does not care for the greater good
Any reliance on commercial entities acting voluntarily for the sake of the greater good rather than in selfish interest has always failed. And that is the normal, default mechanism of capitalism: it is supposed not to rely on ethics but on greed. The responsibility for incentivising or enforcing ethical behavior lies with market regulation, namely elected lawmakers.
The elected lawmakers have been sorting software under the mantle of copyright, and the converged force of copyright maximalists have combined their goals and their market power to lobby for nonsensical things like copyright durations of 95 years after the death of the software author.
There is no point in appealing to companies to do their part to their individual disadvantage. This is something that the lawmakers have to get right, and they are paid off in order not to be interested in that.
[ link to this | view in chronology ]
Re: The free market does not care for the greater good
Red Hat, SuSe, Canonical and the Blender Institute, amongst others, would all disagree with you. Google also gives a lot back to Open Source Software, including employing Open Source developers.
[ link to this | view in chronology ]
Re: Re: The free market does not care for the greater good
RedHat has been driving with a rather sketchy licensing scheme for their commercial distributions: if you redistribute their "GPLed" software versions, you lose support. And it ultimately let itself get bought by IBM (and IBM might choke on Open Source eventually like Sun Microsystems did before it got sucked up by Oracle). SuSE got bought out by Novell and afterwards got passed around a lot. Nobody talks Slackware anymore. Or Mandrake or one of its derivatives.
GitHub runs proprietary software and tries locking its users in an ecosystem that is not free while building upon Git. It's always been a lot more profitable to piggy-back on free software rather than foster it. Whenever free software has not fallen by the wayside, it is on a precarious balancing act between proprietary interests and actual free software work.
You cannot go "all-in" on free software but always have to drive a balancing act, and the reason is that the laws of the market do not care for it.
[ link to this | view in chronology ]
Re: Re: Re: The free market does not care for the greater good
Red Hat has a freely redistributed version of RHEL, it is called CentOS, built from RHEL code, but with different the branding. What Red Hat are protecting is their trade mark, not the code, so while redistributing RHEL itself is not allowed, the same code can distributed by using Centos.
IBM buying Red Hat may have a lot to do with Linux being available for their mainframes, and that they have been contributing to Linux for years now, including employing full time Linux developers. Linux looks to have a lot to do with the future of their mainframes.
[ link to this | view in chronology ]
Re: Re: The free market does not care for the greater good
Canonical, at least in the past, had quite a reputation of not contributing very much to the projects they use. People went so far as to call them a leech (though I say that's unfair, because the "host" isn't harmed—IOW, "copying is not theft"). Have things changed?
[ link to this | view in chronology ]
Re: Re: Re: The free market does not care for the greater good
I think that's a bit of an unfair statement to make since Linux distributions have traditionally all been aggregators. To some degree, Ubuntu is a rebranding of Debian, but Canonical had a number of projects like their Unity desktop and a phone spinoff that they tried to make fly.
The one party that really contributes significantly (though that is really a lot under the radar) is RedHat since it bought up Cygnus, the main workforce actually working on GCC.
RedHat also has significant amounts of kernel and, well, systemd developers under their wings.
I'd not call Canonical "leeches" since they have been very free with their distribution from the start, including sending out free CDs. RedHat's CentOS was an independent company providing unbranded RHEL before RedHat decided to get rid of that nuisance by buying it out.
So the balance between contributing and playing by the rules and spirit really has been different for different GNU/Linux distribution providers, and other examples of major Open Source software tend to have a diverse background as well (I am not sure where the current animosities between LibreOffice and OpenOffice stand right now).
[ link to this | view in chronology ]
Re: Re: Re: Re: The free market does not care for the greater go
More deciding that is was useful, and making sure that it continues. It is still going strong. If you want stability use Centos, and if you want bleeding edge, use Fedora, also supported by Red hat.
[ link to this | view in chronology ]
Re: Re: Re: Re: The free market does not care for the greater go
The point wasn't to shit on Canonical, just to say that it's maybe not a great example of "commercial entities acting voluntarily for the sake of the greater good rather than in selfish interest" (in the context of giving back to open source projects).
[ link to this | view in chronology ]
The #1 reason for those who work on open source to do so is that... it's FUN. It's not specifically that's is open source; it's that you're working on stuff that truly interests you. That's all most programmers need. If your bosses get that, then it's great for you and open source projects. If not, well, short-sighted business folk are hardly a rarity. Open source was already a thing before even Unix was released to the world. Programmers just want to have fun. I can't see that ever being stopped.
[ link to this | view in chronology ]
Or, the more selections there are, and the more products there are, the more likely it is that many [projects will du[locate each other and thay there will be more projects that are so narrowly tailored there will be few people who are interested in them.
[ link to this | view in chronology ]
Hard to get higher than the top of the mountain.
At this point, there are feature-complete Open Source products that cover pretty much the entire breadth of IT, with the exception (and this is a pretty big exception) of end-user applications beyond the Web browser. In many areas (as the article pointed out), the Open Source product is the dominant one.
There will always be new projects, but all the catch-up work is pretty much over.
Except for end-user apps, where the catch-up work has been largely abandoned. I guess it's not interesting enough for the people that would be good at it to do in their spare time, and little incentive for corporations to sponsor enough professionals to make products that are superior to their commercial counterparts. Yes, you can get useful work accomplished with, say, OpenOffice, or desktop Linux, but few could argue that they are actually better (in a getting $hit done sense) for end-user tasks than the commercial equivalents. (In my circle of friends, all of which are IT professionals, half of them veteran sysadmins (so no lack of knowledge here), precisely zero of us run Linux on the desktop; it's all Mac, Windows, and Chromebooks.)
[ link to this | view in chronology ]
Re: Hard to get higher than the top of the mountain.
open sources natural strengths play to servers
If you have the choice between an app that people can easily see what it does and has been validated by countless independent experts as being solid or a black box that won't show it's cards but is easier to use, the risk reward for the black box probably makes sense for the end user but not for the admin.
[ link to this | view in chronology ]
Re: Hard to get higher than the top of the mountain.
Using proprietary software has high risks, especially for user work, as file formats change, and a decade latter the file might be unreadable with the latest version of the software. Increasing use of SAS, carries even higher risks of losing everything that you created, and not being able to work because of a prolonged network outage.
[ link to this | view in chronology ]
Common file formats aren't much of a danger
Yes, if you are using some odd proprietary application, there can be an issue reading it in the future. But for widespread end-user applications, there's not much danger of being stuck. (In any case, a typical end-user isn't going to be in any better shape if their Open Source app isn't usable any more; the file format may be technically open, but that doesn't help if there's nothing to read it with.)
[ link to this | view in chronology ]
Re: Common file formats aren't much of a danger
Open source projects are much better at maintaining file comparability, at least so that latter versions will read earlier versions files, besides which if the source repos are still there, you can build earlier versions of programs. Also there is the option of installing earlier versions of an OS in a VM.
In the open source world, it is rare to lose the ability to work with old formats, and there is quite good support for various proprietary formats.
LibreOffice supports a wide rang of file types, click show to see them. There are also several converters for WordPerfect to LaTex available, covering all the old formats. Unsurprisingly the worse supported formats are those from Microsoft word, but if you have ever looked inside the formats you will have realised what a mess they are,
[ link to this | view in chronology ]
Re: Hard to get higher than the top of the mountain.
"with the exception (and this is a pretty big exception) of end-user applications beyond the Web browser. "
End user applications and (possibly) associated devices typically fall into the category of proprietary closed source software being sold by a for profit business who has little incentive to disclose the inner workings of their product. Some even go out of their way to hamper open source efforts at writing software to interface with same. This is hardly the fault of open source developers.
Perhaps I have misunderstood your comment ....
[ link to this | view in chronology ]
The old days were fun.
Even the Big companies swapped Creators.
They bounced back and forth and everyone learned things.
Then there were the Home made programs to Copy and diagnose the other software. Which led to the Current crop of utilities that were created long ago, in design and how they worked.
Keeping this old hackers alive was a good thing, because they Would learn and teach others How they did something special. How many Anti-virus/anti this/that do you want? There is even a bounty system for them by the Big corps.
But,
Atari went from a creator, to a Buyer and distributor. They find it Cheap to pay off a group that made a good/great program(they think) for a pittance, Then make a fortune from it.
Go look up the history of the creator of Zip. tried to sell it,and no one wanted it as it was ALL OVER THE PLACE ALREADY. Then he died and guess who grabbed it?
[ link to this | view in chronology ]
Woke. The dogmatic woke cult sucks productivity out of every system they enter. The kafkatraps and hostile environment created by sophist subjectivity sucks the fun out, then you lose volunteers.
[ link to this | view in chronology ]