Hacked Florida Water Plant Found To Have Been Using Unsupported Windows 7 Machines And Shared Passwords
from the sigh dept
By now, you have likely heard about the recent hack into a Florida water treatment plant which resulted in the attacker remotely raising the levels of sodium hydroxide to 100 times the normal level for the city's water supply. While those changes were remediated manually by onsite staff, it should be noted that this represents an outside attacker attempting to literally poison an entire city's water supply. Once the dangerous part of all of this was over, attention rightfully turned to figuring out how in the world this happened.
The answer, as is far too often the case, is poor security practices at the treatment plant.
According to an advisory from the state of Massachusetts, employees with the Oldsmar facility used a computer running Windows 7 to remotely access plant controls known as a SCADA—short for “supervisory control and data acquisition”—system. What’s more, the computer had no firewall installed and used a password that was shared among employees for remotely logging in to city systems with the TeamViewer application.
If you're not in the IT space, this is base level stuff. Have your computer systems on operating systems that are under active support and are being patched. That is doubly so for any systems that are critical, or which have access to critical systems. And to not have any client security, such as a local software firewall, on such a machine is IT malpractice. On top of the above, it appears that TeamViewer hadn't been actively used by the staff there for nearly six months. So there, again, was poor administration of the environment, with an antiquated remote access application not being removed from the production environment.
Instead, the save in all of this came from the meatware that was fortunately sitting at the machine and actively watching.
The breach occurred around 1:30pm, when an employee watched the mouse on his city computer moving on its own as an unknown party remotely accessed an interface that controlled the water treatment process. The person on the other end changed the amount of lye added to the water from about 100 parts per million to 11,100ppm. Lye is used in small amounts to adjust drinking water alkalinity and remove metals and other contaminants. In larger doses, the chemical is a health hazard.
Christopher Krebs, the former head of the Cybersecurity and Infrastructure Security Agency, reportedly told a House of Representatives Homeland Security committee on Wednesday that the breach was “very likely” the work of “a disgruntled employee.”
It's a water treatment plant for an entire city. In an era where there is an extreme lack of trust in government, dumb stuff like this acts as a supercharger.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: florida, scada, security, shared passwords, water plant, windows 7
Reader Comments
Subscribe: RSS
View by: Time | Thread
That shouldn't even be an option
Speaking of things that shouldn't be possible you'd think that any system that controls the addition of potentially harmful substances would have a built in upper limit so that it's quite literally impossible to change the settings to harmful levels, I guess all sorts of vulnerabilities are being exposed from this attempted mass-poisoning.
[ link to this | view in chronology ]
Re: That shouldn't even be an option
So I don't work in a water plant. However I could see that "safe" levels might depend on external factors. However increasing it by 100x is probably beyond that. So it sounds there there was no sanity checking of inputs at all (and there DEFINITELY should be some).
Also, why is this system hooked up to any network, ever? I can't think of any justification. Someone just tried to poison (or maybe even murder? not sure what the likely effects of that level would have been) and entire city. This is the sort of reason why "air gaping" is a thing (or rather 'was': is anyone still sane enough to be air gaping their critical systems? they should be, but someone clearly isn't).
[ link to this | view in chronology ]
Re: Re: That shouldn't even be an option
The reason that a SCADA system is hooked to the internet is so the engineers don't have to leave their office to check on things, they can do it from their desk and also receive reports and monitor for any problems outside the set parameters. They can also, if properly programmed, shut the system down or make changes to set parameters remotely, within the tolerances set within SCADA. Properly done, you can't exceed those set tolerances without being on site with a dongle.
Problem here is how it was hooked up, the lack of any sort of security, and piss poor programming of SCADA.
The SCADA system should never have been tied to the internet. It should have been tied to an intranet, which is normally when done right, not accessible from the internet outside without proper security pass through such as Citrix to allow access, not to mention the lack of a firewall and the terrible idea of Teamviewer as a remote.
[ link to this | view in chronology ]
Re: Re: That shouldn't even be an option
Coincidentally, the movie I'm watching right now makes it clear that "gaping" and "gapping" are two completely different things!
[ link to this | view in chronology ]
Re: Re: Re: That shouldn't even be an option
I don't know. Perhaps a 'gape' between the SCADA system and the internet would actually be preferable to a mere 'gap'.
[ link to this | view in chronology ]
Re: Re: Re: Re: That shouldn't even be an option
That's probably where they went wrong. They must've misunderstood and thought they were supposed to "gape towards the internet".
Remember, it's way too easy to get licked when you're gaping, so plug those holes.
[ link to this | view in chronology ]
Re: That shouldn't even be an option
The department pointed out that the system did have maximum limits in place in hardware, and alarms that would've alerted the operators to the change if they hadn't noticed it themselves. It was just that in this case the operators acted so quickly that the additional layers of safety measures never had a chance to activate.
[ link to this | view in chronology ]
Re: Re: That shouldn't even be an option
A human quicker than a computer/sensor? Only if the firmware had a built-in delay before taking action, or at least sending out an alert. Like pointed out in all of the above comments, that would be a designed-in failure just waiting for exploitation.
Time to get Clifford Stoll on the job, and find this bugger - he (or she) is obviously intent on more than just molesting a government system, they're out to cause radical, and possibly irreversible, harm to an undeserving populace.
[ link to this | view in chronology ]
Re: Re: Re: That shouldn't even be an option
"A human quicker than a computer/sensor?"
Yes, according to reports an operator was watching the screen as the cursor started moving by itself, making it obvious that there was some kind of breach before the settings were changed.
[ link to this | view in chronology ]
Re: That shouldn't even be an option
Sometimes levels of chemicals that are unsafe for consumption can be used for cleaning the machinery. But that would normally involve taking the treatment pipeline offline, so a safety interlock that prevents the setting while there is an open path into the supply system might be possible.
[ link to this | view in chronology ]
Re: That shouldn't even be an option
Also not a water plant engineer here, but I believe the much higher doses are used when you need to sanitize a section of the water system, like when a new main has to be installed. So while it might be crazy high for normal use, there could be times where you want/need that sort of level.
[ link to this | view in chronology ]
Re: That shouldn't even be an option
"Speaking of things that shouldn't be possible you'd think that any system that controls the addition of potentially harmful substances would have a built in upper limit so that it's quite literally impossible to change the settings to harmful levels..."
It's not unlikely there were. However, if you can hack the system setting those limits then those limits only really apply to keep fumble-fingered legitimate users from breaking shit too badly.
The real harm here is having a system like that connected to the internet at all.
[ link to this | view in chronology ]
The people of Oldsmar were very lucky, and should be very thankful the meatware was there and caught the hack, rather than just stare at the screen and say "Far out, man!"
Lots (most?) of these kinds of systems are not really meatware monitored at all. They rely on the control system itself to detect problems, and to notify someone if there is a problem. Of course a competent hacker or disgruntled employee could probably disable the self-monitoring and / or notification systems, too.
[ link to this | view in chronology ]
Re: people of Oldsmar were lucky
well, they were not lucky to have an incompetent local city government exercising monopoly control over their critical water supply.
Monopolies tend to slack off on the quality of their products and services.
if this was a private company operating the water plant, there would already be several lawsuits and aggressive government investigations underway against its managers and owners.
But the Oldsmar city bureaucrats will instead get a generous budget increase to upgrade their sloppy computer control systems. Oldsmar residents pay the extra cost for government failure.
Good thing that the government doesn't run our farms and food stores.
[ link to this | view in chronology ]
Re: Re: people of Oldsmar were lucky
As everyone knows, the security in private companies is always flawless, this sort of thing never, ever happens when there's a profit motive. ( https://www.techdirt.com/articles/20201215/13203045893/security-researcher-reveals-solarwinds-update -server-was-secured-with-password-solarwinds123.shtml ) Private companies always run the tightest ships when it comes to safety, they never decide that paying fines is cheaper than fixing problems, even if they cost human lives ( https://www.spokesman.com/blogs/autos/2008/oct/17/pinto-memo-its-cheaper-let-them-burn/ ), and there sure aren't any cases of corporate entities causing massive environmental disasters and used legal loopholes to walk away without suffering any consequences, having shoved the cleanup costs onto the public. ( https://cen.acs.org/environment/persistent-pollutants/Chemical-companies-spar-over-PFAS/97/i36 ) EPA Superfund sites? (https://en.wikipedia.org/wiki/Superfund ) All the fault of the public sector.
And thank god that there's no government control of farming, everyone knows farmers are amazingly responsible, never overusing fertiliser, pesticide or antibiotics and they're always super careful when it comes to the handling of chemicals and waste. Everything would be perfect if we just ran everything like the farmers, there wouldn't be any worry about water pollution as they're all just so responsible. ( https://www.nrdc.org/stories/industrial-agricultural-pollution-101 )
[ link to this | view in chronology ]
Re: Re: Re: people of Oldsmar were lucky
Farmers would never sell lettuce grown downstream from a cattle lot.
[ link to this | view in chronology ]
Re: Re: Re: people of Oldsmar were lucky
As everyone knows, consumers can just switch to the other competing water providers if they are dissatisfied with the security/safety of their local government water utility.
oh wait, economic monopolies don't work that way.
Whole point of government water utility monopoly in 97% of US is that government supposedly is very mucn better at it than those greedy, careless private ompanies that provide all your food and life essentails.
''
[ link to this | view in chronology ]
Re: Re: Re: Re: people of Oldsmar were lucky
The comparison of government to business with the intent of political posturing is a bullshit argument. Private business is no more efficient or ethical. Assholes are everywhere, they are in private business and they are in government.
Range wars were so much fun, lets do it again.
[ link to this | view in chronology ]
Re: Re: Re: Re: people of Oldsmar were lucky
Is it does not make economic sense duplicate all those underground pipes to every building to allow competition.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re: Water district
The employee that set this system up in 2003 retired in 2006 and moved on. Windows 2007????
[ link to this | view in chronology ]
I used to work with SCADA.
It uses what is known as latter logic for it's performance, or at least did in the capacity I used it in. Since health and welfare of the employees as well as the facility and the environment, depended on not only computer controls through SCADA but also physical secondaries as backups, not tied to the computer, it is beyond belief that something such as this was not looked at with safety in mind from the start of the design of how the SCADA system would function at this water plant.
This isn't even talking about the lack of updating the operating system and I suspect if the OS was not updated, neither was the SCADA system. Limits can and were set in the system for us. It took a dongle to change those limits as we applied them to get them outside the set parameters. The dongles were highly controlled and never, ever, left in a computer.
To understand that no firewall and no methods of preventing internet connections from a distance were not in place, is just dumb founding. At least an intranet would have helped so that connections were limited to being within the system.
From the outside looking in, it appears someone thought the budget for IT was not needed beyond the hardware and maybe a battery change every five years or so that the CPU uses as back up in the SCADA system.
[ link to this | view in chronology ]
Re:
Ladder logic was developed in the latter part of the last century.
[ link to this | view in chronology ]
Re:
Did you mean ladder logic? Not with 'T's — but with two 'D's. Ladder.
[ link to this | view in chronology ]
Re:
Ladder logic.... Not latter..... You should have caught that being a SCADA expert and all.
[ link to this | view in chronology ]
stares at the monitor
Oh
Look
My
Shocked
Face
Security costs money that doesn't have a big flashing light that tells you, you are safe.
For the cost of a case of tear gas rounds, they could have updated & secured this, but its not photogenic.
Now that the bad, that they were told could & would someday happen, has hit there will be a big panic that will result in blaming the IT dept of 1 for the city not funding basic security needs of the water supply while making sure every officer has 5 repurposed military medals for bravery of killing a 12 yr old with a water balloon using the bomb removal robot.
Its just water, not like its important.
The real danger is that PoC might forget their place.
[ link to this | view in chronology ]
Re: Water troubles
There are like, 50,000 independent water districts in the US. Some large and comprehensive and some with like 200 users in the Sierra Nevada or Montana. No way will these archaic small systems going to be able to have "meatware" sitting there 24/7. But basics like firewall and 2FA should be solidly in place, even with ancient windows.
[ link to this | view in chronology ]
Re: Re: Water troubles
It seems that firewalls were disabled on the OS, so no real excuses there, the tech is built in. From what I understand, if you're using the paid for Teamviewer versions, it's quite easy to tie it in to active directory, giving controlled access to whoever needs it and making it easy to revoke permissions from individual users. There's no real excuses here, except the usual trend of people using shortcuts and cheap options and not taking security seriously until after they're breached.
[ link to this | view in chronology ]
Re: Re: Water troubles
I can see the necessity for remote indicators for alarms and warnings, but to allow complete control remotely is just plain stupid.
[ link to this | view in chronology ]
Hope they find the perp. There is no place in society for those who would do such a thing.
[ link to this | view in chronology ]
Florida. Amerika's penis.
[ link to this | view in chronology ]
no one should be using windows 7 when windows 10 is free ,it sounds like theres a limited budget for it staff, everty state, county has its own software,
even with windows 7 you could whitelist ips,
As we see every day there seems to be no basic standard of security on government owned pcs.
no mandatory standards as regard firewalls, os updates etc
no ip adress outside this list can acess our network.
And who sets up a pc without even installing a firewall .
[ link to this | view in chronology ]
Re:
The problem with windows 10 is that you're constantly getting updates which you cannot possibly verify. And the free versions involve forced rebooting for those updates, which sounds like a bad idea when it comes to the systems responsible for water quality...
These computers need to not be connected to the Internet in the first place.
[ link to this | view in chronology ]
Re: Re:
"The problem with windows 10 is that you're constantly getting updates which you cannot possibly verify. And the free versions involve forced rebooting for those updates..."
Windows has matured to the point where it's now great for a good many things. This is not one of those things.
For a plant like this, if you intend the controlling device to do one thing very well and remain online for twenty years without a hitch, you install Linux or BSD.
[ link to this | view in chronology ]
Re:
To be able to upgrade windows, also requires that the SCADA control software can be run under windows 10. It get expensive quickly if you also have to replace you SCADA controllers to get control and development software supported by a later operating system. Not being able to run required software for some external system is one reason why there are still XP systems in use.
[ link to this | view in chronology ]
Re: Re:
This.
You can't expect some group that would not even bother to properly secure the system in the first place (No firewall. Really?) to perform updates. Let alone if said upgrades could cost money.
Windows 10 is not a viable upgrade for production systems that need stability. The only version that is, is their LTSB (Long Term Support Base) edition and that is only available through an enterprise level subscription agreement with Microsoft. Even if you have one of those subscriptions, they severely limit how many of LTSB installations you can activate with your subscription, and you cannot get more.
Never mind that the upgrade to Windows 10 will normally break old production software. Many production software suppliers will charge for an OS upgrade patch. Either due to legal / certification / support requirements, or just greed. Even when they don't charge money, many of those things that use external hardware require drivers that won't function on newer versions of Windows and have no updated drivers available. In some cases an OS upgrade could require an organization to upfront the cost and downtime of an entire brand new replacement system.
The responsibility for upgrades does fall on the organization for performing them, but the bigger issue is the fact that the industry itself uses EOL upgrades to force recurring payments against organizations that cannot pay, with the general public soldering the risk when they don't. It's an unaddressed problem that's existed for decades. I guess enough people haven't been killed yet...
[ link to this | view in chronology ]
Re:
No one should be using Windows when Linux is free.
FTFY
[ link to this | view in chronology ]
Re: Re:
Does this cities SCADA software run on Linux? If not, then you're just replacing one expensive problem with another expensive problem. While there are F/OSS SCADA solutions (I just did a simple Google search and found three plus a site that claimed to have eight listed without even scrolling), they would still require funding to implement and train their employees.
[ link to this | view in chronology ]
Re: Re: Re:
"Does this cities SCADA software run on Linux?"
Does it run on Windows 10?
[ link to this | view in chronology ]
Re: Legacy Machines
I've got a bit of 1995 hardware I use for part of my software production process. When I need it, I boot up the Win7 machine (last one to support it); the manufacturer is long out of that business.
Industrial machinery is like that -- the non-computing hardware isn't upgraded every third year whether it needs it or not.
Given the recent software infrastructure attacks, I'm wondering how long before my main software development machine will get an air gap from the internet. I had that back in 1998 with a certain graphics chip development company -- one machine for internet, the other machine for the actual work.
[ link to this | view in chronology ]
Re: using windows
Or if you want security, how about Linux or one on of the BSDs?
[ link to this | view in chronology ]
Re: Re: using windows
Back east here, management is scared of anything on a desktop that's not Windows. And my open source is not without its own headaches, including security.
Meantime, I need to ship a product and can't shut everything down.
Claim:
There's a market for a stuxnet-proof route across an air gap with provable trust properties. Jump drives, with the OS auto-execing special files, don't quite do it.
In the import direction I have hardware design and software development software, and the supporting datasheets and 3-D models. OS updates are not accepted.
In the export direction, I have binaries and design packages, such as I might send to a PCB fab and/or assembly house.
[ link to this | view in chronology ]
Re:
"no one should be using windows 7 when windows 10 is free "
You know how I know that you haven't considered the many problems with windows 10, which might not have retained compatibility with some legacy software being used?
The other criticisms are fine, but there certainly are reasons not to update Windows even though it's "free".
[ link to this | view in chronology ]
Shut down this shitty fucking website.
[ link to this | view in chronology ]
Re:
How about fucking off if you don't like it?
[ link to this | view in chronology ]
I also have no less than 3 XP machines, all running just fine, TYVM. Two of them are in daily production, the last is kept as a spare, just in case ('cause they're all old). The CNC machinery they operate will run on Win7, but why bother - if it works, don't mess with it.
Oh, yeah... they are both connected to the world only via sneaker-net.
I might be repeating myself, but so what, it never hurts to hear it again: When it comes to connecting to computers outside of your immediate physical control, you must adhere to the Prime Maxim of security - Practice Safe Hex! If you depend on someone ele's soft/firm/hardware to protect you, then you've already lost, you just don't know it yet. Actually THINKING about security - there's no acceptable substitute.
[ link to this | view in chronology ]
Missing the relevant part, "hooked to the internet."
Even with the latest Windows 10 and full security measures, it's not "if" but "when."
My favorite client has two separate physical networks; wires, routers, machines. One internal business network, one for internet stuff. Each machine on the internal network has a cron script that periodically tries to ping half a dozen different IP addresses on the internet. If it ever gets a response, it sends a signal and the entire network starts doing an orderly shutdown.
There are orange cables and orange Ethernet ports. And there are white cables and white Ethernet ports. Every new employee is told that there will be extreme management displeasure if anyone decides to plug a cable into a non-matching port...
Secure? Not perfectly; they still have to move documents, spreadsheets, and CAD drawings from one side to the other with thumb drives, but way better than "install an antivirus and hope for the best."
There's no reason for a public utility's control systems to be hooked to the internet. All of the "explanations" boil down to laziness and/or incompetence.
[ link to this | view in chronology ]
Re: Or Budget in 2017
Or someone who is the IT person moonlighting from day job as secretary at the local church.
[ link to this | view in chronology ]
99% of the time, compatibility issues with the new OS are the reason for not upgrading - especially in systems like this as they tend to rely on very specific hardware to function.
[ link to this | view in chronology ]
Re:
More than that -- the later the version of Windows, the worse the hardware observability and controllability. And where did my parallel port go!
[ link to this | view in chronology ]
But, but
The same? vintage gear (voting machines though) Used in GA, NJ, SC, DE, Louisiana.
[ link to this | view in chronology ]
SuperBowl
There's an aspect to this story that's largely been ignored: it happened two days before the SuperBowl, only 10 miles from the stadium.
I've only seen the story covered by publications that are either tech focused or local to Florida. The SuperBowl angle means it deserves broader coverage. It elevates the situation from unlucky or random hack to potential serious terrorist attack. The Windows 7 thing? It's not good, but it's not surprising, either.
[ link to this | view in chronology ]
Re: SuperBowl
I'd say it's worth holding back until we know exactly what the perpetrators were actually trying to do and for what purpose, which should come out in the eventual criminal prosecution.
Is it concerning and something to be aware of during the investigation? Sure. But, we could all use less sensationalist reporting and it's not going to be productive to hype this angle up when other much more mundane possibilities exist.
I'm sure everyone else will be jumping on this as soon as it's proven that it was a targeted attack on the Superbowl, if that was the case. But, until that evidence comes out, it's actually sort of nice not to have the worst case scenario being breathlessly speculated upon by people with no expertise on the matter, without the facts to back that up.
"It elevates the situation from unlucky or random hack to potential serious terrorist attack."
It's doubtful that luck was involved, but also less likely to be a terrorist attack than the current theory of being a disgruntled ex-employee. What we know about the attack thus far implies that terrible security practices meant that anyone with access to that password would have been able to do this from wherever they were located. Teamviewer is generally considered secure with good password and login management, and I'm not aware of any major security flaws that would have allowed people access without knowing the password.
So, the poor security discount luck as being a major factor, while the fact that anyone working for the company in certain roles would have had access without a terrorist motive, and some people so really dumb things when they feel they've been wronged by an employer. Let's see what the evidence says before jumping to a conclusion other than the one provided by Occam's Razor.
[ link to this | view in chronology ]
There are places still using XP. Frys is still using XP on there machines around here and Kaiser Permanente is still using Windows 7
[ link to this | view in chronology ]
Right now it appears that a disgruntled insider, perhaps a former employee, accessed the system with the shared password in the normal way and made the changes.
This is the typical 'cybersecurity' threat. Not a foreign hacker using elite hacking skills, but an insider using the system as designed in a malicious way. The application appears to have been set up for only console access, then someone set up a RDP system for remote desktop access and shared the password.
Of course this story is immediately being used to push other agendas. The San Jose Water Company (NYSE:SJW, a large private utility company) is justifying their latest rate increase requests by the need for increased cybersecurity.
[ link to this | view in chronology ]