Heart Surgery Stalled For Five Minutes Thanks To Errant Anti-Virus Scan
from the death-by-horrible-IT-support dept
If you've ever had the pleasure of simply asking one medical outfit to transfer your records to another company or organization, you've probably become aware of the sorry state of medical IT. Billions are spent on medical hardware and software, yet this is a sector for which the fax machine remains the pinnacle of innovation and a cornerstone of daily business life. Meanwhile, getting systems to actually communicate with each other appears to be a bridge too far. And this hodge podge of discordant and often incompatible systems can very often have very real and troubling implications for patients.For example, one patient recently undergoing a heart transfer had the procedure interrupted for five full minutes after a PC connected to an essential piece of monitoring equipment began a scheduled anti-virus scan:
"According to one such report filed by Merge Healthcare in February, Merge Hemo suffered a mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer. Fortunately, the patient was sedated, and the doctors had five minutes at their disposal to wait for the computer to finish rebooting, start the Merge Hemo application again, and complete their procedure without any health risks for the patient."Fortunate, since "death by shitty hospital IT support" doesn't sound like a particularly fun way to go. The filing with the FDA by the company in question (Merge) notes that the blame was the fault of the hospital's IT support, who ignored software instructions that state the folders being used by Merge's software should always be whitelisted from any anti-virus platforms:
"Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors' computer. The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly."Here's the thing: aging systems and shoddy medical IT support are the least of the medical industry's problems. The biggest problem continues to be that medical technology security remains little more than an afterthought, leaving underfunded IT support frequently outgunned. That has resulted in a major wave of ransomware attacks that in some instances have actually forced hospitals to revert to using paper only while they get sorted out (underfunded school systems have been having a dramatic uptick in similar attacks).
And as Internet of Things companies push hospitals to embrace even more sophisticated technologies, you can expect things to get worse. After all, this is a sector that can't even secure doorbells, refrigerators, thermostats or even tea kettles. What could possible go wrong as these technologies are introduced into an already marginally-competent medical IT sector?
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: anti-virus scan, computers, heart surgery
Reader Comments
Subscribe: RSS
View by: Time | Thread
Instead of reporting the problem, that is sloppy for safety critical software.
[ link to this | view in chronology ]
Re:
The antivirus would finish with that file in a matter of seconds at most. The reboot took minutes. That could, in some cases, be the difference between life and death.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
On one hand, yeah, that's kind of stupid of the IT guys.
On the other hand, that's kind of stupid of the Merge developers, and extremely stupid of them to not fix the problem when it's brought to their attention.
[ link to this | view in chronology ]
Re:
Many times you can't patch the systems (even the OS), because it'll void the support on the system. Install AV? yup, can't do that either. partition it off from the network it so it can't talk to the internet? Nope. Not allowed, vendor won't support it if it can't phone home.
But when the thing comes down with malware or crashes in the middle of a procedure, it's somehow always IT's fault for not "doing things right".
It never seems to come back to procurement, for not engaging IT on the system, or on the vendor for using undocumented (and unsupported) operating system API's to tie into custom hardware that requires crash-prone 3rd party licensing app to read a crypto key off of a thumbdrive.
Medical device software is pretty much crap. In many cases, even if IT had the authority to do so (and they usually don't), they couldn't fix the issues these systems see.
[ link to this | view in chronology ]
Re: Re:
Well there's your problem. That should be an immediate "No way". Why spend money on software when the company is that stupid?
OF COURSE the computer that runs the heart surgery program should never EVER be connected to a network. Yikes.
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
IT doesn't get a say-so in it. Doctors who know little to nothing about software and computers listen to vendor sales pitches, make the decisions and then order IT to do it. Of course, when it goes to shit, IT gets the blame, not the doctors. Shit rolls downhill.
[ link to this | view in chronology ]
Re: Re: Re: Re:
Maybe the hospital or doctors were talked/ bribed/ convinced to get this software from this vendor.
And if the vendor knows they have a monopoly on the market, will they do their best to make robust software? Or will the developers say "There's no chance a virus scan would interrupt the process, but even so, let's just tell the users not to do it instead of adding code to handle the error".
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
A PC?
Who the hell thinks it is OK to use a PC to run an essential piece of medical equipment during a heart transplant? I assume said PC was running windows, poorly secured and connected to the hospital network which also was connected to every other PC in the building . . . what could go wrong!
[ link to this | view in chronology ]
Re: A PC?
[ link to this | view in chronology ]
Re: Re: A PC?
Guess what it says in the EULA about things like that? It's one part of such documents worth minding, instead of treating these things like the appliances they still are not.
[ link to this | view in chronology ]
Re: Re: Re: A PC?
[ link to this | view in chronology ]
Re: Re: A PC?
Doctors like Windows. It's like what they have at home.
[ link to this | view in chronology ]
Re: A PC?
That said, it would be smart to have a second machine running the same disc image ready to go in case of an emergency like this.
[ link to this | view in chronology ]
Re: A PC?
Whitelist? No. After installing a operating software, all the system should be rechecked,what if their were a virus from the materials installed? Or an update change a permission, that's what the it guy is for. They are supposed to be knowledgeable about the system, and the programs in use. There could be other problems,
[ link to this | view in chronology ]
Re: Re: A PC?
[ link to this | view in chronology ]
Re: Re: A PC?
When was the last time you heard of virus scan crippling a Linux system?
Yeah, that's what I thought. Talk about lame.
[ link to this | view in chronology ]
Re: A PC?
Having it configured so that processes like virus scanning can happen while it is actively in use is a serious problem, but not one that is special to Windows PCs.
[ link to this | view in chronology ]
One correction
We can do the job - if we're given the manpower, the authority, and the tools needed.
Also, faxes are used because people in charge (not your IT sector) often think they are MORE secure than email. Yes, that's wrong, but "Everyone has a fax" and "I don't know the email of the person these are going to!" are the battlecry of the "Email is too HARD!" crowd.
It's not like IT has veto authority in these matters.
[ link to this | view in chronology ]
Re: One correction
[ link to this | view in chronology ]
Re: One correction
This machine shouldn't be actively used...so why in holy hell is is actively running AV scans every hour?
Nightly, at most, would suffice. I mean an AV scan would drastically eat up system resources. And as a system ages could it even finish an AV scan in the time it would take before another one starts?
Constant AV scans are idiotic and counter-productive, especially on machines that are not used to browse the web.
[ link to this | view in chronology ]
Re: Re: One correction
Time sensitive, organs are.
[ link to this | view in chronology ]
Re: Re: One correction
The second mode is "on access" where individual files are scanned as they are read/written. This is the area where you would want to exclude the files requiring real-time access, unless you're using a really fast drive.
The third mode is "heuristic" where files aren't really being scanned at all; it just watches the memory and flags up if anything suspicious is seen.
Now some AV software has this nasty habit of displaying a dialog when something suspicious is found, potentially locking things up on the system until a decision is made. This is bad AV design.
But in this case, the AV software was being run every hour, which, as you say, is the IT staff not knowing what they're doing. There is absolutely no reason to do an on demand scan every hour. This shouldn't even really be an option. For quality AV software, it will see that other processes are under high utilization, and throttle back the scanning to not impede other operations... and then the scan will likely take more than an hour to complete, resulting in a cascade of scheduled scans.
[ link to this | view in chronology ]
Re: potentially locking things up on the system
It’s the kind of thing that used to happen with single-tasking OSes in the 1980s. Not a modern multitasking OS that is supposed to be fit for the 21st century.
[ link to this | view in chronology ]
Re: Re: One correction
[ link to this | view in chronology ]
Re: One correction
Lot to be said for competent IT not trying to 'excuse' their way out of doing a job, too. (I did the work on my own time, after hours...what was their excuse?)
[ link to this | view in chronology ]
Re: Re: One correction
In my experience, the problem IT departments face is twofold: first, they don't directly generate revenue. There's no line you can point a bean-counter to that says "here's the value to the company". This means that they are often viewed as a drain on resources that is to be minimized, rather than the essential utility that it actually is.
Second, if an IT department is excellent and doing its job properly, then there will always be clashes and people pissed off at them -- particularly management, because much of their interaction time will consist of raising holy hell in opposition to some stupid idea or another.
It means that being good at IT is as much a political thing as a technical one. Setting up a new network copier is technically easy, but that kind of thing is often littered with various political mines.
Bad IT departments just give up on the political battles and do the minimal amount they are required to do to keep their jobs. You can spot these pretty easily -- the people in these departments just look defeated and cranky.
I have immense respect for good IT people. I wouldn't last a month in their shoes.
As an aside, when I am evaluating a company that I'm unfamiliar with, the three most valuable things I can learn to get an idea of the company's character are what the custodial staff, the secretarial staff, and the IT staff think about how the company runs.
[ link to this | view in chronology ]
Yes, the software license probably says "don't use this for important systems" but the software could still be written to provider more user friendly options. Stories like this are too common for it always to be an IT department problem.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
The problem comes down to using a device that is inappropriate for the task at hand. Trying to define extra rules to compensate for the inappropriateness is an ugly workaround, not a fix.
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Just give it time
[ link to this | view in chronology ]
it's just a flesh wound...
While the anti-virus scans your machine,
To do so is folly and you won't be too jolly
When blood starts coming from places unseen.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Just have to add this
https://xkcd.com/463/
[ link to this | view in chronology ]
Re: Just have to add this
Thank you.
[ link to this | view in chronology ]
IT kills - news at 11
Software that doesn't recover? Nonsense. Is it really better if the software continues to run while saying, "I can't tell if the patient is dying!" for the next 5 or 10 or 20 minutes?
Automation that runs "whenever". Scans of folders that shouldn't be. Sounds like typical "blanket IT policy" to me. Regardless of what I'm doing, for example, software install will pop up and kindly give me two minutes to shut everything down before the reboot...even if I'm not there.
An accident/near-accident like in the article happens every so often...that a piece of software using Windows (or not Windows) fails or is clobbered by some automated behavior and critical functionality is lost or someone's life endangered. (Cases I can recall on Windows: A software install that endangered a patient in South Africa, I think; a blue screen reboot of a mission-critical device on an aircraft carrier.)
If your software is really critical, it needs to be treated that way. The sad truth is: that is not where we are. No one gives a care about security or reliability...just so long as the software package is secure and reliable enough to make it through the sales demo without crashing. Since no one buying software looks behind the grandiose GUI interface, it's going to stay that way.
So in the end, this case is hardly worth mentioning: "Patient might have died, news at 11." And if not that then, "Hospital settles wrongful death lawsuit, news at 11." Actually, these days, neither event would make the evening news.
[ link to this | view in chronology ]
Re: IT kills - news at 11
It shouldn't simply crash, IMO. The developers should have forseen the possibility that their software might not be able to access the data for some reason, and programmed it to be robust enough to deal with that.
"Release it now, the damn users can beta test it" isn't really acceptable for medical applications.
[ link to this | view in chronology ]
Re: Re: IT kills - news at 11
The software was locked out of something it needed access to, in order to do its job. Locked out, whether it spends the next 20 minutes saying "Unable to connect to patient; retrying" or simply crashes outright...
...IS NOT RELEVANT TO THE PATIENT. Crash or retry: the patient is equally at risk.
[ link to this | view in chronology ]
Re: The software was locked out of something it needed access to...
Linux, for example, does not do this. So a background scan need not block higher-priority activities from proceeding at the same time.
[ link to this | view in chronology ]
Re: Re: Re: IT kills - news at 11
[ link to this | view in chronology ]
Re: Re: Re: IT kills - news at 11
I've encountered medical software which, through system reboot and reinitialising the programme, can take ten minutes (although clearly this one is a bit faster than that) to be fully up and running.
I stand by what I said, the software should be robust enough to deal with it. And since Merge said the file folders should be whitelisted, they obviously knew it was a problem.
As an aside; a problem was discovered with the Alaris (also sold under IVAC and Carefusion brands at different times) Signature volumetric infusion pump. It was known as key bounce, and what could happen was that a keystroke would inadvertently be registered twice (the keypad flexed slightly, so two distinct contacts could be made without the button being fully disengaged).
Clearly, if this key bounce would happen while setting the infusion rate, a rate could be entered that was ~10x what it should be (e.g. 99.3ml/hr instead of 9.3). It totally did happen of course.
Now, Alaris said - quite rightly BTW - that the user should have checked what rate they'd programmed before pressing start. But in the end, Alaris were forced to roll out a software upgrade which detected two key presses within a very short time, and gave a warning message and an audible indication.
So anyone who works for Merge, hold onto your ankles 'cos you might still get your balls felt.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
It's not even that. Some doctors and hospitals simply refuse to do it or work with the other party. But hell, i remember doing this stuff manually.
Getting the other doctor(s) / hospital to actually pay any mind to those records they wanted so badly? Whole other fucking ball game.
[ link to this | view in chronology ]
Typical
[ link to this | view in chronology ]
Re: Typical
In fact, I think some systems are still on XP...
[ link to this | view in chronology ]
IT Department:
NEVER SCHEDULE AV scans on ANY computer. Defender (MSE) will tell you when it would like to scan WITHOUT interrupting your usage.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
I Thought Windows Was A Multitasking OS ...
Even if disk contention is a problem, a rationally-designed OS like Linux offers this feature called ionice, which lets you reduce the I/O priority of the scanning task to minimize disruption to more important activities.
I use this for backup tasks, for example.
[ link to this | view in chronology ]
Insecure by design
Malware writers praise Merge for providing a safe place to hide their stuff!
How can you ensure something is free of malware if you are not allowed to look everywhere malware might be?
[ link to this | view in chronology ]
Mission-Critical Software
Apps might be a problem, but I've found that most software is available, at least a functional match, for Linux or BSD. Custom software can be ported over to Linux, at least if the author is still available. Worth looking into.
[ link to this | view in chronology ]
Re: Mission-Critical Software
Do you think this problem would have occurred if a "good tech" was available? Oh, right, companies don't pay for those..."Mordac the preventer of information services" is cheaper.
[ link to this | view in chronology ]
RE Hospital IT
I pulled out my phone and no service.
No problem I have WiFi calling I'll just hop on their guest network.
I connect and it wants me to enter my phone number so they can send me a text to gain access.
I still haven't figured out if this is by design or incompetence.
I suspect somewhere the CIO is telling the rest of the suits that they no longer need a guest network because hardly anyone ever uses it.
[ link to this | view in chronology ]
I'm Confused!
[ link to this | view in chronology ]
NBD
to the List :drug mixups/toxicity, hospital 'food', patient name confusion,
hospital acquired infections++...no wonder
the medical establishment is the 3rd biggest killer:
http://www.health-care-reform.net/causedeath.htm
http://www.cnbc.com/2016/05/04/medical-errors -are-third-leading-cause-of-death-in-united-states-study.html
Medical errors are third-leading cause of death in United States: Study
Dan Mangan
4 May 2016
Go to the doctor or hospital when you're sick in the hopes of getting better, and you might end up dead, instead.
A new study estimates that medical errors are actually the third-leading cause of death in the United States, responsible for a whopping 251,454 fatalities in 2013.
Only heart disease and cancer, which respectively killed 611,000 people and 585,000 people that year, outpaced medical errors, according to the study published in the medical journal The BMJ...
[ link to this | view in chronology ]
Stupid is as stupid does
[ link to this | view in chronology ]
Re: Stupid is as stupid does
Agreed, but it seems doctors would rather lobby for protection from such lawsuits.
[ link to this | view in chronology ]
Re: Re: Stupid is as stupid does
we are proud to be a Microsoft house.
Seriously.
[ link to this | view in chronology ]