Dependencies: Both Technological And Human, On Display In The Story Of A Developer Who Deleted Code Being Used By ICE
from the our-interconnected-world dept
Three years ago, we had a pretty fascinating story about how a developer, after getting an ambiguously threatening note from a company about how a bit of his code might violate the trademark of another company, deleted all of his code from NPM (Node Package Manager), a key repository for node.js code. One of the bits that the developer deleted (totally unrelated to the potential trademark dispute) was simple code that tons of websites relied on -- leading many of them to break in response. The story raised all sorts of interesting questions not just about trademark, but namespaces, who controls code, dependencies, and much more. Indeed, the story was so interesting to me that I (very loosely) used it as inspiration for a science fiction story I recently wrote that will be released very soon (more on that very soon as well!)
Having been thinking a lot about all of that lately thanks to the story I was working on, I was surprised to see a similar situation pop up last week, with slightly different issues. This one involved an IT automation company, Chef, that helps lots of organizations better manage the configuration of various physical and virtual servers. The story kicked off with some controversy as someone noticed that Chef had signed a contract with ICE. Lots of people got (reasonably) angry about this, following on a pattern that has been playing out in the tech sector over the last few years.
Chef's CEO put out a pretty lame email and blog post, basically saying "but we signed this deal under the previous administration," which (among other things) fails to recognize that ICE was pretty fucking terrible during the previous administration as well.
But here's where the story gets a lot more interesting. A former Chef employee named Seth Vargo, who had created a bit of open source software called Chef Sugar, got quite reasonably upset to learn that ICE was using his code to more efficiently detain children.
"I was having trouble sleeping at night knowing that software—code that I personally authored—was being sold to and used by such a vile organization," he told Motherboard in an online chat. "I could not be complicit in enabling what I consider to be acts of evil and violations of our most basic human rights."
Vargo asked the company to explain this (prior to that awful blog post mentioned above) and then, after a few days went by without a response, took down his code from two key repositories: Github and RubyGems. As he wrote:
I have removed my code from the Chef ecosystem. I have a moral and ethical obligation to prevent my source from being used for evil.
Of course, because no one has learned anything, multiple other systems depended on that code being in those repositories, and those systems started breaking as well. Even more fascinating, some of the people who this caused problems for still supported Vargo's decision:
props to the Google Engineer who yanked code from Chef for working with ICE. You've made my job harder today, but I really don't mind.
— marea rosa (@smrt_fasizmu) September 19, 2019
This certainly started getting much wider attention -- leading Chef's CEO to issue an update on Friday, which first seemed to unnecessarily attack Vargo:
On Thursday, September 19th an action was performed by a trusted community member in violation of the standards of open source software (OSS) development. The individual yanked several RubyGems that they authored while employed by Chef. In order to remove the gems, they first removed the other owners and took unilateral action to yank the gems, violating established processes for making OSS changes and improperly removing property which Chef owned. This ownership has been established through the Github history of commits, licenses, etc. The individual did not have Chef’s permission to remove these items from the RubyGems site.
So, obviously, some may point out that since Vargo's work was initially done on Chef's dime as an employee, he has less ground to stand on. But, again, as an act of protest, it's pretty fascinating.
Also, it turned out to be incredibly effective. By Monday, Chef had completely reversed its position and said that it would not renew its work with ICE:
As many of you know, we began our work with the U.S. Government in earnest in 2014 and 2015. This included DHS and its various departments under a different set of circumstances than exists today. The overarching goal was to help them modernize their computing infrastructure and create a cooperative community of IT professionals inside the government that could share practices and approaches in a similar way to many open source communities. Policies such as family separation and detention did not yet exist.
While I and others privately opposed this and various other related policies, we did not take a position despite the recommendation of many of our employees. I apologize for this. I had hoped that traditional political checks and balances would provide remedy and that our relationship with our various government customers could avoid getting intermingled with these policies. However, it is clear that checks and balances have not provided relief to the fundamental issues of the policies in question. Chef, as well as other companies, can take stronger positions against these policies that violate basic human rights. Over the past year, many of our employees have constructively advocated for a change in our position, and I want to thank them.
After deep introspection and dialog within Chef, we will not renew our current contracts with ICE and CBP when they expire over the next year. Chef will fulfill our full obligations under the current contracts.
The company also promised to donate the equivalent revenue that it had received from the contracts to charities helping people impacted by ICE's family separation policy.
This whole story is quite interesting on multiple levels. Seeing tech workers recognize that they have some moral stake in how tech they develop is used is quite amazing -- especially given the exaggerated (and incorrect) stereotype that Silicon Valley never cares or thinks through these things. That's never been true, but it's especially interesting to see people taking some element of ownership over how what they've developed is eventually used. Second, it's another interesting example of how interdependence on code hosted elsewhere is creating a somewhat fragile web in certain places. I'm almost surprised that we haven't seen this as an attack vector -- gaining control over repositories and doing something with them that impacts lots of other services.
Either way, it's a representation of how interconnected the entire world is -- at both a technological and human level.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: code, dependencies, ice, open source, seth vargo
Companies: chef
Reader Comments
Subscribe: RSS
View by: Time | Thread
Having read Techdirt for many years I am finally beginning to understand why there is a block of people who out an hate Mike.
What Mike needs to do is make a pass through Phnom Penh and Auschwitz and see what the meaning of some of his more stupid ideas results in.
[ link to this | view in chronology ]
Re:
"I am finally beginning to understand why there is a block of people who out an hate Mike"
Is it because every one of them is a voice in your head?
[ link to this | view in chronology ]
Delusional
How about you point to where Mike supports concentration camps and genocide?
[ link to this | view in chronology ]
Re: Delusional
It's simple logic - he hates Mike for whatever pathological reason, he's been told by his right-wing echo chambers that "socialism" is a label to apply to everything he hates whether relevant or not, therefore Mike writing an article about code library sharing means he's the same as Stalin.
It's easy once you forget those pesky things like facts, logic and sanity that this guy happens to be immune to.
[ link to this | view in chronology ]
Re: Delusional
My guess is that those Asian countries don't have copyright troll representation. Dumbasses like him consistently believe that a lack of copyright is equivalent to crimes on the level of arson and murder, you see.
[ link to this | view in chronology ]
Re: Re: Delusional
Judging from the type of comments that they are (like this on, generic, unrelated to the article or only barely related) I would guess it's more likely just astroturfing of some sort.
[ link to this | view in chronology ]
Re:
The fuck?
[ link to this | view in chronology ]
Re:
Why is this a random comment based on nothing in this post?
Can someone relegate this person to the history section??
[ link to this | view in chronology ]
OK, although I work with devs and GitHub to some degree this confuses me, possibly because I don't really get involve with Ruby on the coding end. Maybe someone can help clarify here?
"In order to remove the gems, they first removed the other owners and took unilateral action to yank the gems... The individual did not have Chef’s permission to remove these items from the RubyGems site."
So, how did he remove them? Unless I'm mistaken about the way gems operate, surely that would require some admin level access to the project? Did someone at Chef mess up and leave him as an admin, did he retain some sort of access due to some Google involvement in the project, or am I missing something?
[ link to this | view in chronology ]
Re:
RubyGems (the server infrastructure) has basically 2 permissions levels for each gem, the general public that can download the gem, and the gem's owner that can do anything with the gem. You can add and remove owners via their command line or api by providing an email address.
It's fairly common in open source projects to allow any trusted developer to have fairly broad rights and different projects have different standards for trust.
I know few developers that created an addon for a larger project, they get contracted by the company that owns the larger project to add some features to the addon, or develop some related addon, and then after the contract expires they continue to maintain their addons. Because there was a contract some or all of the addon code it could be owned by the company, but because of the prior relationship it's in everyone's interest that the developer can still work on the code.
[ link to this | view in chronology ]
Re: Re:
That makes sense. It still sounds amazingly open to abuse to me, but that's probably me coming from a sys admin background toward dev ops having had a solid background in desktop user support early in my career - I have definitely been taught to trust no-one with access rights!
[ link to this | view in chronology ]
Re:
More or less. While Chef owns the copyright on the code, Vargo owns the accounts that provided it. This was a chain-of-custody problem; Chef should have hosted the code in its own repositories, not relied on a third party to control them (not even when he was an employee, and certainly not after he was no longer with the company).
[ link to this | view in chronology ]
Re: Re:
"While Chef owns the copyright on the code, Vargo owns the accounts that provided it."
Yeah, there's your problem. I've worked for more than one company where the entire organisation was in danger of going down the tubes because they didn't care much for access control and ownership until the day they realise they don't own their own domain name, or the head dev's mate who was controlling their AWS account decided not to pay the bill when something went south on a personal level...
Such is the way of things at the moment, I fear - rush to get things out the door, then forget to make sure you have everything locked up behind you.
[ link to this | view in chronology ]
Re: Re:
With git, all developers, and quite a few users, will have a clone of the master git repository, git being designed to be used with multiple copies of the code, and so it is easy to duplicate the Github version under a new name. Ownership of the public repository is more a matter of convenience than someone controlling the existence of the code.
[ link to this | view in chronology ]
I can understand the ICE hate...
But does it need to reach this level? Its methods are frequently deplorable, but ICE does serve an important mission of the US government.
If the court of public opinion has decided that hamstringing ICE at every opportunity is the best way to make it reform its practices in regard to immigration and detention, we should be discussing and preparing for the unwanted side effects that are likely to occur as a result. These boycotts also affect necessary enforcement functions against things like drug smuggling and sex trafficking.
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
So what is the proper level of response to an organisation that put children in detention camps where some of them dies?
Why shouldn't a private person take action when his morals and ethics isn't in agreement with the aforementioned organisation and its practices?
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
"ICE does serve an important mission of the US government"
The mission isn't the problem, it's the inhumanity committed in the name of doing it. It might be important to keep streets free of litter, but you'll still be criticised if standard procedure were to tase and jail anyone spotted dropping a plastic bag.
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
LOL the objective is to abolish
[ link to this | view in chronology ]
Re: Re: I can understand the ICE hate...
LOL!
If you think the US government is ever going to just walk away from the nation's borders and throw them open to anyone who wants to walk/fly in, you're nuts.
Abolish ICE and another agency will be created immediately to do what ICE does. It'll just have a different name.
[ link to this | view in chronology ]
Re: Re: Re: I can understand the ICE hate...
We already have agencies that can do that job. Why did, do, and will we need another one?
[ link to this | view in chronology ]
And that is a mission the U.S. government was carrying out well before ICE and the Department of Homeland Security were ever a thing. The U.S. doesn’t need ICE or DHS to handle immigration and customs enforcement — and we don’t need their apparent inhumanity towards anyone who isn’t White.
[ link to this | view in chronology ]
Re:
ICE is just INS with a different name. It's been around since the early 20th century.
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
Don't worry, the Trump administration already cut the funding to fight sex trafficking.
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
ICE was created by the PATRIOT Act after 9/11 in a manner that left it nearly unaccountable to any meaningful oversight. When most people say "Abolish ICE" they mean roll back to the pre-9/11 state of things.
[ link to this | view in chronology ]
Re: Re: I can understand the ICE hate...
I'm not going to quibble over the word 'most' but a significant number of people who advocate abolishing ICE literally don't want any border enforcement whatsoever. They don't believe there even should be borders in the first place, let alone enforcing them.
[ link to this | view in chronology ]
Re: Re: Re: I can understand the ICE hate...
"a significant number of people who advocate abolishing ICE literally don't want any border enforcement whatsoever"
This is where you cite your source for those figures...
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
Do you think that Judge Dredd is satire or a documentary?
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
Bruce.
Something many have not seen yet, and its on the Wiki..
Look up Gov. Police agencies.
ICE and DHS are now in charge of over 40 of the policing agencies.
They were HIRED, to do a certain job, and are going above and beyond that. ALSO, its interesting that, as I understand it...They WERE HIRED.. anyone remember the Names they have used??
blackwater?
[ link to this | view in chronology ]
Re: I can understand the ICE hate...
It's definitely not necessary for drug smuggling to be illegal, and may not even be helpful. People die due to high American drug prices; others import from Canada in amounts that are technically illegal.
[ link to this | view in chronology ]
Nice one
An excellent article, Mike. I hadn't see the support from those inconvenienced. Solidarity at its finest.
Do we know which license the Ruby Gem code was published under?
[ link to this | view in chronology ]
The ICE is melting
With all of the hate against the ICE, I think it’s begun to liquidate.
Yeah... I suck at these things. It’s 8:40 a.m over here in Pennsylvania and I’m only half awake.
[ link to this | view in chronology ]
I will laugh if ICE sues everyone involved in the violations
I assume a contract states what is and isn't allowed to happen that might cause problems. A former worker signing in, changing lots of settings and then deleting work done while under contract seems like it could cause millions of dollars worth of liability and damages. If the hacker (since they were not authorized to access the work this way any longer, it is a CFAA violation) isn't made an example, it will happen again and again. Good luck moron. You just screwed up your life over a morally questionable action.
[ link to this | view in chronology ]
So by deleting his code, he's causing ICE to be less efficient at detaining children. Therefore, it's somehow a win that ICE keeps right on doing the things he doesn't like, but now causes even greater harm by its reduced efficiency? Which is more likely: removing this code causes ICE to stop detaining children, removing this code causes ICE to process detainees less efficiently (thereby keeping them detained even longer, possibly under even worse conditions since the reduced efficiency may increase overcrowding), or removing this code is a purely symbolic gesture that hurts a bunch of unrelated organizations and doesn't impact ICE's detention practices at all?
Yes, I saw that Chef will not be renewing their contract. ICE is a big organization, and there's nothing here that suggests Chef's non-renewal will affect ICE's family detention subunit in the slightest. It might do that, or it might only impact other ICE missions that enjoy broad support.
Wouldn't it be funny if the only things this hurts are the naturalization/visa issuance divisions, groups that help immigrants become lawful residents/visitors?
[ link to this | view in chronology ]
Hey, Maz, ever heard of CHINA? Have any of its products?
First, don't forget the suicide nets at Foxconn factories making your precious APPLE phones. That wasn't enough to give you pause, just another PR problem.
Experts Call on UN to Investigate China's Killing of Religious Dissidents for Their Organs
https://www.theepochtimes.com/experts-call-on-un-to-investigate-chinas-killing-of-religious-d issidents-for-organ-harvesting_3094703.html
https://www.theepochtimes.com/c-organ-harvesting-in-chin a
Now, as the only true humanist here who's actually concerned about even furriners, isn't secretly a rabid militarist in favor of "humanitarian bombing" for Oil and Empire nor a corporatist intending to destroy the US of A precisely because of our freedoms, I've been railing at trading with murderous Communists and especially the worst, China for longer than Maz has been alive.
YOU are betraying your own and American principles, Masnick. You advocate for Chinese-model total surveillance / "social credit" score society here, all done efficiently by unlimited untaxed corporations which decide who has a public voice.
You are so inconsistent that are a total fraud.
[ link to this | view in chronology ]
Re: Hey, Maz, ever heard of CHINA? Have any of its products?
I bet you don't give away your supposed science-fiction novel! You'll try to extract money or email address up-front, not risk AFTER people have read it.
I further bet that you don't actually EVER release it! You may have showed it around to your pals who will of course say nothing even less than fullsome praise, because they're all frauds too!
But to EXPOSE yourself (WAY more than your dry vaguely factual basis here) and skills to people not highly biased in your favor? HA. You are not a good writer to start with, and your notions are childish ones from the covers of lurid 50s magazines, like Elon Musk.
[ link to this | view in chronology ]
Where’s your book, bitch?
[ link to this | view in chronology ]
Re:
I would also like to see your aerospace company
[ link to this | view in chronology ]
Re: Re:
How's that Lincoln Bandlow defense fund coming along, bro?
[ link to this | view in chronology ]
Re: Re: Hey, Maz, ever heard of CHINA? Have any of its products?
Why would he want our email addresses? And how does writing a piece of fiction expose him any more than writing on this site already has?
[ link to this | view in chronology ]
Delusional man spews falsehoods, get cited facts and runs off
Hello dishonest coward, when are you going stop running away when confronted by facts?
[ link to this | view in chronology ]
Re:
But... but... whatabout?
Let's ignore the poor quality of your information, but did you ever consider actually defending some of the crap Trump has done rather than try deflecting the conversation elsewhere? Is the problem that you know it's indefensible?
[ link to this | view in chronology ]
By the way, the premise of imprisoning children and...
ripping families apart, and letting children die: that's FALSE. You minor and gullible liberals are eating the dog food that others create. And of course there's foreign serf "PaulT" hating America as always.
NO? CITE IT, THEN.
[ link to this | view in chronology ]
Re:
Right after you cite how getting America's army to pay for the border wall is the equivalent of getting Mexico to pay for it.
[ link to this | view in chronology ]
Delusional man spews falsehoods, get cited facts and runs off
https://www.nbcnews.com/politics/immigration/24-immigrants-have-died-ice-custody-during-trump-admini stration-n1015291
https://www.newsweek.com/ice-director-immigrant-parents-ultimately-responsible-sep aration-children-1454115
Now go away, or I shall taunt you a second time!
[ link to this | view in chronology ]
To me this is more about stupid business practices.
I do not want to take away from the moral choices being made by the developer with what I'm about to say.
To me, this issue speaks volumes about how commercial companies are abusing open source tools for monetary benefit and the stupid decisions they are making.
In this particular case we have a commercial entity with their product, Chef, relying on the personal source code repository of an individual developer for key functionality of their product.
Why would any clear thinking business owner decide to rely on the resources and services of an outsider, provided for free by that outsider, to supply a crucial resource needed for their business?
We just saw the result of that decision.
If you are going to run a business based on providing reliable services to support open source software why would you not make sure that you can continue to provide reliable service in the event that the projects that you rely on for the software shut down or remove themselves from public access?
[ link to this | view in chronology ]
Re: To me this is more about stupid business practices.
In my experience, it probably wasn't that at all. Devs sometimes decide to take the path to least resistance at the moment they need something, not really considering consequences long term outside of their code. Many a time I've seen some random personal accounts being used to host mission critical code because the developer needed to bypass some restrictions that they felt were slowing down development work.
In other words, it's not greed, it's people not keeping tabs on what others in their team are doing until they need to work out a problem. Fine for early dev work, but not something that belongs in production - but someone has to remind them, I think.
[ link to this | view in chronology ]
Re: To me this is more about stupid business practices.
Git based development results in multiple copies of the code repositories, where a developers copy only differs from the 'master' in the changes they have made, and which have not been pulled into the master repository, and other copies updated to copy the changes. So while that deletion made a statement, it should not have been more than a minor inconvenience to chef, as they likely have several copies on machines within the company.
[ link to this | view in chronology ]
Chef says:
That's an inaccurate simplification.
The Open Source Definition bans discrimination against persons, groups, or fields of endeavor. That means that, for example, a license that says "this software may not be used by ICE" is not an open-source license. There are such licenses, and Bruce Perens has discussed why they violate the principles of open source, even if they're well-intentioned:
When Licenses Discriminate
Sorry, Ms. Ehmke, The “Hippocratic License” Can’t Work
However, there's nothing in the Open Source Definition that says you have to keep code you've previously worked on available, on your own private accounts, if you no longer wish to support it. The Open Source Definition says that if you distribute a program, you have to make its source code available. It doesn't say that you have to continue distributing that program. And, not for nothin', Vargo is neither the owner nor the legal author of the programs in question; the company that employed him when he wrote them, Chef, is.
Chef relied on a third party, who no longer worked for the company, to provide access and distribution of code that it owns. That was unwise. Even if you believe that what Vargo did was unethical (and I don't), it wouldn't have impacted Chef if Chef had taken proper custody of its own codebase.
[ link to this | view in chronology ]
Re: if you distribute a program, you have to make its source co
No, that is not part of the Open Source definition as such, that is an additional requirement commonly known as “copyleft”. It is possible to have Open Source software distributed under licences with no copyleft requirement.
[ link to this | view in chronology ]
First:
How, exactly, can source code violate a trademark? A copyright I can understand but a trademark? Source code is never seen by the public and is never exposed as trade dress so it couldn't possibly cause any kind of brand confusion.
And second:
The choice for a software house always boils down to "build or borrow?". They can either invest a lot of time, effort and money into building a tool, widget or feature OR they can use an appropriately licensed open (or sometimes closed) source solution. The foreign code rarely behaves exactly as you would have written it but it's "good enough" and get incorporated into your project to save time and money. This is and has been happening globally for a lot of years. It's not a new phenomena and it been consistently causing problems, sometimes as this article describes, since it began.
This practice has also long been recognized, though not nearly as broadly as it should be, as a security issue. Vulnerabilities are often found in such code and every outfit using it needs to update to the latest version or find some other workaround to the security hole. The worse situation is that an "owner" of the foreign code could intentionally introduce a vulnerability or even get his account hacked resulting in vulnerabilities added to the code. None of this is a new story either and has happened many times. It has to be considered in the cost vs benefit analysis of using external code.
The only bit of any of this that is new is why the code was taken down, not that it was taken down.
[ link to this | view in chronology ]
Re:
I've trademarked my company name, which is "Error."
If your program displays any sort of dialog box which uses that word, your code is infringing upon my trademark.
[ link to this | view in chronology ]
Re: Re:
A valiant effort but sorry, no banana.
[ link to this | view in chronology ]
I have no opinion on whether he should have done this, but as someone who has fought numerous battles trying to get people not to have build systems depend on being able to just go download from the Internet in the middle of a build, my biggest gripe with this is that it didn't cause wider fallout. People need to learn to cache dependencies locally, and having things suddenly break due to a dispute between parties you don't know is a nice motivator for that.
[ link to this | view in chronology ]