Contractors Who Built Healthcare.gov Website Blame Each Other For All The Problems
from the nice-try dept
With all the problems associated with the Healthcare.gov rollout, a bunch of fingers (including ours) pointed at the usual list of government contracting cronies who built the thing. The deal was done under an existing contract (so no open bidding) and involved the same "usual suspects" who have been connected to a number of other large government computer systems debacles. Not included anywhere in the list were companies with experience building large-scale web services -- which you'd think would be helpful here. However, in testimony before Congress, the contractors are insisting that it's not their fault. CGI Federal was the main contractor behind the site, and Cheryl Campbell, a senior VP from the company, is in charge of trying to point fingers elsewhere, mainly at the Centers for Medicare and Medicaid Services (CMS), which CGI Federal says was in charge of the actual building of the site.CMS serves the important role of systems integrator or "quarterback" on this project and is the ultimate responsible party for the end-to-end performance of the overall Federal Exchange.Basically: it's the government's fault. We just build the damn thing. If they didn't tell us to build the right thing, or test it properly, well, it's their fault. Also, someone else we won't name is really at fault:
Another contractor was awarded the contract for the Data Services Hub portion of the Federal Exchange.Oh, and also another unnamed contractor:
The first set of issues for users dealt with the enterprise identity management (or EIDM) function provided by another contractor, which allows users to create secure accounts.Of course, it's not too difficult to figure out who the "other" contractor is. Because it's on the panel too. QSSI built the Data Services Hub and the "EIDM" functions mentioned, and QSSI is owned by Optum, whose executive vice president Andy Slavitt is testifying as well. And, you know, it's not his fault. First, he insists that the Data Service Hub worked splendidly throughout, no matter what anyone else might say. EIDM, of course, is having some trouble, but that? Why, other vendors are to blame there too:
It is relevant to note that the EIDM tool is only one piece of the federal marketplace’s registration and access management system, which involves multiple vendors and pieces of technology. While the EIDM plays an important role in the registration system, tools developed by other vendors handle critical functions such as the user interface, the e-mail that is sent to the user to confirm registration, the link that the user clicks on to activate the account, and the web page the user lands on. All these tools must work together seamlessly to ensure smooth registrationIn other words, if only those other vendors did their job right, the whole thing would work much better. Oh yes, also someone (nameless) decided to change the specs at some late date:
It appears that one of the reasons for the high concurrent volume at the registration system was a late decision requiring consumers to register for an account before they could browse for insurance products. This may have driven higher simultaneous usage of the registration system that wouldn't have occurred if consumers could "window shop" anonymously.The final note, going back to CGI Federal, is to remind Congress that building websites is really hard.
Unfortunately, in systems this complex with so many concurrent users, it is not unusual to discover problems that need to be addressed once the software goes into a live production environment. This is true regardless of the level of formal end-to-end performance testing -- no amount of testing within reasonable time limits can adequately replicate a live environment of this nature.That's true to some extent, but it doesn't excuse many, many of the overall problems with the system, which did not appear to be built with any recognition of how to build a high-traffic transactional website. While CGI Federal would like to point fingers at everyone else, it was its name on the contract, which it received through questionable means, and it should take at least some responsibility for it. Perhaps, if it was so "complex," it shouldn't have taken on the job.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: blame, contractors, healthcare.gov
Companies: cgi federal, optum, qssi
Reader Comments
Subscribe: RSS
View by: Time | Thread
J. K. Rowling approves.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
> to build a working site, sounds like it might
> be time to give the job to some other company,
> or at least put the offer of such on the table.
Someone (or multiple someones) need to go to jail over this whole thing, not just lose a contract. Instead, the same people who already flushed a half-billion dollars of our tax money down the crapper are going to get even more money to fix it. And the government bureacrats in charge, like Sibelius, instead of losing their jobs, apparently "have the full confidence of the president", at least they do according to his spokeshole, Jay Carney.
And they *really* need to stop trying to minimize their gross incompetence by calling the problems with the Obamacare website "glitches" and "technical snags".
A complete catastrophic failure of a system built on 10-year-old outdated web tech is not a "glitch" under any commonly accepted definition of that term.
[ link to this | view in chronology ]
Re: Re:
ASCII and ye shall RX.
The path to jail will be via the removal of the copyright notice and that is due to fraud. The fraud would be getting paid for that Open Source code.
[ link to this | view in chronology ]
This could have easily been avoided -- at very low cost
Among the people who support it are numerous programmers, testers, system analysts, system admins, network architects, etc.
It would have been much cheaper to simply have the feds manage the project, open-source the entire thing, and let those who support it perform the development.
Yes, yes, yes, I know that has its drawbacks: but this cost a frigging fortune and IT DOESN'T WORK. How much worse could it possibly be? And not only that: this monstrosity isn't open-source, therefore it's (a) insecure and (b) insecurable. (I trust everyone is aware that security by obscurity never works, therefore closed-source code is insecure by its very nature. Anyone who isn't aware should undertake remedial education in security fundamentals.)
Some very impressive projects have been built in this manner -- and many of them have demonstrated the ability to scale radically when required.
[ link to this | view in chronology ]
Re: This could have easily been avoided -- at very low cost
Your point about all code having errors, bugs, and security mistakes is true for any reasonably large project. Open Source says help us make the code better by inviting others to help and submit code patches.
[ link to this | view in chronology ]
Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: This could have easily been avoided -- at very low cost
when we ALL KNOW, it's a Democrat Vs. Republican argument, Republicans are due for a win, 'anytime now'.
Its not like the republicans are skill at picking fights they can win or anything !
At least Mr Masnick makes ONE person supporting them, again, they cant even agree on anything amongst themselves.
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: This could have easily been avoided -- at very low cost
Plus open sourcing the software is only part of the battle. This system needs to push/pull information from multiple databases (not open source btw, loaded with personal information) from multiple agencies. That is an integration problem that open source would not have been able to solve.
I hadn't actually heard yet that this was a no-bid contract. That was a serious error. This project was too big in scope to be slid in under another contract. That is a key driver in how and why its failing spectacularly. It was a huge development effort, but from the contract side is was viewed as a trivial add on.
I, know, its shocking to refer to a $400 million project as trivial, but that seems to be how the contractors treated its development and integration.
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
Open sourcing this would certainly not have guaranteed success, and the code itself is only one small part of the scope of the project, but... the code portion of this is certainly something that can be handled through the OSS model. Similar problems, on a similar scale, are handled with OSS code every day.
Personally, the problem with this project has nothing whatsoever to do with whether the code is open source or not, and everything to do with incompetence on the part of the contractors. Open sourcing would not have affected that.
[ link to this | view in chronology ]
Re: Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: Re: Re: This could have easily been avoided -- at very low cost
[ link to this | view in chronology ]
Re: Re: This could have easily been avoided -- at very low cost
Nobody is saying they should donate their time. Ask all the Red Hat employees.
Learn the difference between free speech and free beer.
http://lmgtfy.com/?q=free+speech+not+free+beer
[ link to this | view in chronology ]
Re: This could have easily been avoided -- at very low cost
Ummmm...the feds were managing the project...managing the way they always do. To failure.
[ link to this | view in chronology ]
Integrator failure
[ link to this | view in chronology ]
Re: Integrator failure
[ link to this | view in chronology ]
Re: Re: Integrator failure
So what they are saying might actually be RIGHT, and what TD is saying is probably WRONG.. Ahh ok, thanks for clearing tha tup for us.
[ link to this | view in chronology ]
Re: Re: Re: Integrator failure
[ link to this | view in chronology ]
Looking for a new job?
https://www.usajobs.gov/GetJob/ViewDetails/352656900
[ link to this | view in chronology ]
Re: Looking for a new job?
Er, no. Every government contractor always has job openings for IT specialists.
[ link to this | view in chronology ]
Re: Re: Looking for a new job?
[ link to this | view in chronology ]
Re: Re: Re: Looking for a new job?
[ link to this | view in chronology ]
Re: Re: Re: Re: Looking for a new job?
There is a second site, governmentjobs.com, that is for any government job including city, county and state job openings. Once in a while a federal job is posted on governmentjobs.com, especially hard to fill jobs.
[ link to this | view in chronology ]
Re: Looking for a new job?
[ link to this | view in chronology ]
Re: Looking for a new job?
[ link to this | view in chronology ]
Excuses
2 There is no accountability: no one cares if it doesn't work. The contractor still gets paid.
3 The contractors built what was on the spec. It's not their fault the spec was wrong and incomplete.
4 Obama-care is a tax, not a viable health care benefit. The government will get their tax.
[ link to this | view in chronology ]
1. You can not build it right if it is not designed right.
(Constructions firms do the building; design firms do the designing and rarely are the two the same firm.)
2. Design firms can not design it right if the design specifications are wrong.
(Design firms do the designing; project management firms make the specifications and rarely are the two the same.)
3. Project Management firms can not write the correct specifications if the owners do not know what they want and differences of opinion of what it is they want.
(The owners of the health care system is the American people.)
It is impossible to make a system work when there is no conscientious on what it is the system is suppose to do.
If one wants to understand how big a debacle the health care situation is one really needs to return to another era around 1900 and recall how Americans viewed alcohol.
Across the nation there were saloons. Saloons were not like present days bars. There was alcohol, gambling, and whores. Killings, robberies, and sexual disease were common combined with a living standard so low that some of today's more repugnant areas of degenerated third world poverty stricken cities would appear prosperous by comparison.
As this unsavory section of town was unacceptable to be elitists of the day a reform movement was launched. The 18th was passed ushering in the most violent ear of American existence since the bloody days of the Civil War. Streets of major cities rain red with blood from mob hits.
The debacle formally ended with the passage of the 21th amendment.
More relevant the Republican is still hated for the elitist know it all that produced the 18th amendment and the single minded psychology of religious fundamentalists controlling government.
This is not going to end well for the Democrats.
But it is going to take some time. It took 65 years plus for the Republicans to reach the depth of hatred they had in the 1930 and they still have not recovered anywhere near their status of 1900.
Also, health care will be just as completely different in the future as alcoholic and saloons are today (from 1900).
[ link to this | view in chronology ]
Re:
Guess you've never been to a Bikers Bar.
[ link to this | view in chronology ]
You are missing the real target here
[ link to this | view in chronology ]
It's time for the Blame Game! (or is it Blame-Gate?)
[ link to this | view in chronology ]
Hakers and cyber-terrorism claims...
[ link to this | view in chronology ]
Its the "free" money, not the product
This may be similar to Solyndra where a company gets a so-called "loan" not to actually produce anything useful, but to get "free" taxpayer $$$$.
Since the website developers may have been paid for "work" already done, will bankruptcy be the next step?
[ link to this | view in chronology ]
Re: Its the "free" money, not the product
[ link to this | view in chronology ]
'admit nothing, blame everyone else'!
[ link to this | view in chronology ]
The truly sad part...
[ link to this | view in chronology ]
Re: The truly sad part...
[ link to this | view in chronology ]
A different perspective
CMS is notorious for requirements that leave out most of the details. Imagine that you raise goats in your back yard and need a fence to keep them in. If CMS were writing the RFP (request for proposal), it might ask for "a fence" and when pushed to specify what kind of fence, they might answer "we really like picket fences". Then after you quote them a picket fence and start building it, someone at CMS would ask... "Ummm... will this keep the goats in?". And when you ask "why didn't you put that in the requirements", you are answered with "you should have just known"....
And no, I'm NOT making this up - I've been in the meetings where this exact reasoning was used.
Oh, and I do know that on this particular project at the state level, requirements that were needed to hit the MANDATED Oct 1 implementation date weren't finalized on Oct. 1 by CMS. Of course it's not all CMS's fault - they have to do what congress tells them to do even if it's impossible (in many cases congress sets the implementation date without any input from the folks that are going to implement it).
[ link to this | view in chronology ]
Re: A different perspective
This problem isn't limited to CMS (or even the software industry) by a long shot. It's everywhere.
A part of the problem is with the contractors who accept these vague requirements and offer up bids on them without actually knowing what the work entails. An RFP that just says "build a fence (we like picket)" is obviously trouble to begin with and shouldn't be bid on without further clarification.
Any contractors willing to bid on something like that are, themselves, being irresponsible in that it is literally impossible to know what to bid. Good contractors would just chuckle and throw the RFP in the waste basket.
This disaster is the result of problems in the industry across the board. Despite what people seem to think (this is an example of government screwing things up), this is actually something that happens frequently in the purely private sector as well. It's just easy to ignore it until it happens to a multimillion dollar project.
[ link to this | view in chronology ]
Re: Re: A different perspective
[ link to this | view in chronology ]
Re: Re: Re: A different perspective
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Optum
I am suprised to see Optum added to the list of government cronys because of their past legal problems including going before congress to justify themselves. My guess is someone's palms somewhere have been greased with $$$.
[ link to this | view in chronology ]
A suitable contactor must exist somewhere
Oh, that's why the government wouldn't want to talk to them.
Or vice versa. Maybe they don't want to work on a job mired down in bureaucracy and politicians.
[ link to this | view in chronology ]
Welcome to launch day!
They build entire worlds that get populated by MILLIONS of players in A SINGLE DAY! (all of whom need to register and set up payment)
OK there's some lag, maybe a que to wait in a while, but most have it down to almost a science these days.
[ link to this | view in chronology ]
Re: Welcome to launch day!
[ link to this | view in chronology ]
LEARNED LONG AGO..
I find this hilarious.
3-4 companies all doing PARTS of a program..
THEN SLAP them together and say it works..
WITH OUT TESTING??
The States did this years ago, for State Medicaid..SO WHATS THE PROBLEM??
[ link to this | view in chronology ]
Shorter Obamacare
[ link to this | view in chronology ]
All's Well That Ends Well.
The ObamaCare form is not as good as the Internal Revenue Service's downloadable tax forms, which supports fill-ins with Acrobat 6.xx. When I do my taxes, I start with a text file containing the information for the previous year, make a copy, modify it to reflect this year's information, paste numbers into the downloaded Acrobat file, and print it off. Most of the information does not change from one year to the next, so this is a quite economical proceeding. From the IRS's point of view, the advantage of fill-ins is that they are machine-readable, being printed in a designated font and a designated color. As against these technical advantages, the ObamaCare form is good for five years rather than one year.
To my mind, a paper form is the obvious and appropriate way to hand something like taxes, or real estate transactions such as mortgages and leases, or high-stakes insurance. "Put it down in black and white," they say, in other words, put it down on paper. The web site was an exercise in people playing at being techno-modern in the first place. That was why it proliferated useless JavaScript. The Web works reasonably well for buying small articles on Amazon, with an average value of perhaps ten dollars each. The Web is even better for giving books away, gratis, to the whole world, to anyone who can be prevailed on to read them. You just convert everything into Acrobat or HTML, put it up on your website, create indexes, and you are in business. The Web tends to get into difficulties when you try to use it for things involving serious amounts of money.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Guess where the job was actually done
[ link to this | view in chronology ]