The City Of Baltimore Blew Off A $76,000 Ransomware Demand Only To Find Out A Bunch Of Its Data Had Never Been Backed Up
from the nice-work-if-you-can-get-paid-to-do-it-and-then-not-do-it dept
The City of Baltimore was hit with a ransomware attack in May of this year. Criminals using remodeled and rebranded NSA exploits (EternalBlue) knocked out a "majority" of the city's servers and crippled many of its applications. More details didn't surface until September when the city's government began reshuffling the budget to cover the expenses of recovering from the attack.
The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing.
Johnson, who also serves as the city's chief digital officer, received significant criticism from local authorities for the response to the May 7 attack. City council members alleged a lack of transparency and communication in the wake of the incident, as well as an inability to maintain a functional organization "during an emergency event." He also also never drafted a continuity of operations plan for an IT attack of the kind that occurred.
It looks like the list of stuff Johnson was being paid to do that he never did. Hence the catastrophic outcome when the city refused to pay the $76,000 ransom. Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain.
City residents should be asking WTF their tax dollars are being spent on. The city's audit of its compromised system rolls on, delivering even more embarrassing details about the city's IT skill set. (via Ars Technica)
A new audit of Baltimore’s information technology department says the agency lost key data during May’s ransomware attack because some in the agency used an outdated method for storing files: the hard drives on their individual computers.
[...]
“Performance measures data were saved electronically in responsible personnel’s hard drives,” [Baltimore City Auditor Josh] Pasch reported. “One of the responsible personnel’s hard drive was confiscated and the other responsible personnel’s selected files were removed due to the May 2019 ransomware incident."
Bureaucracies are prone to understatement and the assessment of the ghastly state of affairs by Pasch was no exception. According to Pasch, the permanently-missing data resulted in a "loss of confidence" in the city's IT department's ability to do its job.
This understatement brought the hearing to a halt as council members expressed their disbelief that city data was not being backed up. Their comments were less understated.
Hearing that, City Councilman Eric T. Costello, a former government IT auditor himself, stopped the hearing.
“That can’t be right? That’s real?” Costello asked.
It's apparently real. City data needed for an audit cannot be recovered because the IT department never made an effort to express the dangers of storing the only copy of data locally. It also apparently never made a push to create cloud backups of important files. When the ransomware struck, the stuff locked up was -- in far too many cases -- to only copy of that stuff.
The tragically hilarious postscript to this is the city's response to Ars Technica's request for info on the city's cyberattack recovery plans.
Ars has requested information from the city regarding the contracting details for the recovery, but the city has thus far provided no data. Requests for data on the status of patches and disaster recovery plans were refused because the documents do not exist as a result of the ransomware attack.
It's easy to mock governments for their inability to properly handle the massive amounts of data they collect, create, and retain. And so we shall. The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you're going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: backups, baltimore, frank johnson, ransomware
Reader Comments
Subscribe: RSS
View by: Time | Thread
'Always back up your stuff': A good idea no matter the scale
I see the idea that everyone needs to learn to back up their stuff the hard way applies to government agencies as well as individuals. Just a pity that people keep having to learn that lesson the hard way, rather than learning from those that came before.
[ link to this | view in chronology ]
Re: 'Always back up your stuff': A good idea no matter the scale
"The Cloud" will let people learn it in a new hard way, when they get banned from a service and find there's no customer support, or get told that a service is shutting down in a few weeks.
[ link to this | view in chronology ]
Re: Re: 'Always back up your stuff': A good idea no matter the s
"The Cloud" - another name for "somebody else's computer".
[ link to this | view in chronology ]
Re: Re: Re: 'Always back up your stuff': A good idea no matter t
"Dude, I'm gonna turn my computer in a couple weeks, ya better get yer shit off by then"
—various cloud services, paraphrased.
[ link to this | view in chronology ]
We we're saving money!!
Backups are just ways to burn money!
It'll never happen to us.
[ link to this | view in chronology ]
Re:
If they had a competent IT guy it would have only cost the amount for a few large hard drives and the electricity to download some FOSS backup solutions.
[ link to this | view in chronology ]
Re: Re:
But would a competent IT person have been able to create this linked-in profile?
https://www.linkedin.com/in/frank-johnson
Its comically absurd.
[ link to this | view in chronology ]
Re: Re: Re:
LinkedIn completely checks out:
Leads all digital transformation programs and supports the Mayor's ongoing efforts to modernize the City of Baltimore's IT capabilities, which also include scaling the local IT ecosystem to drive awareness & tech investment in Baltimore City.
Now the city has to modernise and invest. Job done.
[ link to this | view in chronology ]
Re: Re: Re:
Awesome link. Other than all the tech buzzwords, what I see from his profile is that the guy was and still is a salesman. In my experience it is super rare to find a salesman with a firm let alone deep understanding of the tech (in whatever field).
[ link to this | view in chronology ]
Re: Re: Re: Re:
Probably be President someday.
[ link to this | view in chronology ]
Re: Re:
It takes far more than a few hard drives and some FOSS software to back up all of the data generated by every employee of every agency for a city the size of Baltimore. The data storage costs alone would be massive, and constantly growing.
Some other asshole probably thought it should only take very little effort and resources too, so that's what IT got instead of a proper budget and staff, and that's how the city ended up where they are.
[ link to this | view in chronology ]
Re: Re: Re:
Once again, I said competent IT guy. Most of the information could likely be reduced to text and backed up with a greatly reduced memory footprint.
Water bills, power bills, typed police/fire/ems reports ect.
Some information may not be able to have their text extracted and reduced like audio and visual recordings. You only need to save image producing files (i.e. there website formatting or fire report formatting) once also. They don't run youtube or something. The fact is hard drives are huge now and a few of the large ones could likely backup the city of Baltimore.
You should probably not apply for IT positions.
[ link to this | view in chronology ]
Re: Re: Re: Re:
Here's 100TB hard drive if you don't believe me.
https://www.zdnet.com/article/worlds-largest-ssd-hits-100tb/
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
You arent in IT obviously, backing up to a 1tb drive, yea thats how it works, 200 bucks all ya need. You have no idea of what a backup structure look like on an Enterprise scale just STFU people like you are why things like this happen.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re:
Apparently you're a hostile idiot. No competent IT person, again, would backup a hard drive image rather than compressible data in that circumstance. You are truly an idiot.
[ link to this | view in chronology ]
Re: Re: Re: Re:
I work for a large city the size of baltimore.
You don't know how hellish it is trying to deal with their data. And it all needs to retain in original formats. Help me
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
What is your specific problem? If I was doing a cheap semi-lazy (but competent I hope) job of it I wouldn't bother writing scripts to extract the text and then having the document reconstituted into the proper file structure when the data had to be restored. That would lead to really compressible data but it might be a much larger workload for programmers than you have budget for.
A cheaper and easier solution that's much lighter on programming is to get a backup solution that downloads required files from pre-specified folders on a schedule (monthly/weekly whatever). Obviously it is a good idea to stagger backup schedules so nothing gets overloaded in the process.
Finding and running the best compression algorithm for the expected data-type would be my next step. Here is a quick set of replies for the question "what is the best compression algorithm?"
https://www.quora.com/What-are-data-compression-algorithms?redirected_qid=18888451#
As some of the answers note there is no one best compression algorithm for every file so some research and experimentation to find good algorithms for the file-types you are backing up may be necessary.
You would have to get the city departments to cooperate by dropping their work product into the right specified folders and you may have to investigate their computers to find the correct folders to backup too but that can easily be made part of the job. It's not too difficult though it may be time-consuming for a large city.
FOSS solutions on the market already have the ability to just download a folder instead of imaging an entire drive. You can also make them just download specified file types in a folder if that is more optimal.
I would also create 2 backup server/computers that run on alternative schedules so if one set of backups is corrupted you have a good chance of recovering from the other, even if is slightly older. You would have a better chance of being able to recover most of the data.
These are just general thoughts on the issue. If you have a specific problem I would give you my "I hope somewhat competent" opinion if you desire.
[ link to this | view in chronology ]
Targets...
The fact that so many municipalities have paid out ransoms recently has painted a huge target on every hospital, town, and county government. They made the right call in not paying the ransom!
Some insurance companies are just "Insuring" by paying the ransom, causing the problem to spiral downward.
Now, not backing up. (Or backing up on TAPE), that is the wrong decision.
[ link to this | view in chronology ]
What I don't understand is this: I have backed up my entire home network since the 90s, and use a dual NAS solution plus rotating offsite backups for over a decade. This entire solution cost me a total of around $400 over 10 years ago, plus remembering to take the offsite drive with me when I go to the bank. And I've got redundant backups of all data on all devices on my network.
Surely an IT department can manage at LEAST this level of data security for their endpoints?
[ link to this | view in chronology ]
Re:
Back when I worked in IT, I created a backup solution that merged and deduplicated all files from all endpoints on my networks, and it included a web interface where individuals could use their AD logins to recover any files backed up on endpoints that were under their management. The backup repository could be stored anywhere, as it was encrypted. The software to do so was free; it just took me a few hours to configure and add to local policy for administrative file access.
Essentially, this will work for any device on a network that allows network access. It also would have been resistant to a ransomware attack, as it was a one-way system. New writes just get deduplicated; deletions required direct access, which means admin account credentials to a locked down system.
[ link to this | view in chronology ]
What backup system did you use?
That sounds like a nice solution! I am not aware of one with those features. Would you please share some info on it?
[ link to this | view in chronology ]
Sounds to me that the extortionists need to dramatically increase the amount they demand.
[ link to this | view in chronology ]
Re:
I'm afraid you might be correct.
[ link to this | view in chronology ]
I'm sorry, but I must respectfully disagree.
Some portion of that $6 million (or the estimated $18 million) may be going to data recovery: pulling things off backups, rebuilding data from hardcopy (dead tree data), or going back to original sources and polling institutional memory when no other method works.
But the rest of it, to "harden" city systems? That's pure technical debt. Money that should have been spent and hadn't been. Systems that are not vulnerable to the latest exploit don't come about by themselves, they get maintained - patched, backed up, put on new hardware when necessary. Security processes get evaluated and updated. Backups get made. Worst case scenarios get gamed out.
If you wish, consider the 'rebuilding' cost as the cost of a hard security audit. And that cost, as a whole, is likely more expensive than if they'd put a security audit into the budget in the first place.
[ link to this | view in chronology ]
oh no it's the racisssseses
they to blame
dindunuffin
[ link to this | view in chronology ]
Re:
That's the problem...the IT department dindunuffin to prevent this from happening in the first place.
[ link to this | view in chronology ]
FYI: “Dindunuffin” is a racist term.
[ link to this | view in chronology ]
Re:
Everything is racist to leftists
[ link to this | view in chronology ]
Re: Re:
And everything is fair game to a Trumptard.
[ link to this | view in chronology ]
It is when the term was originated/popularized by bigots using it as a racist slur.
[ link to this | view in chronology ]
Re:
You realize, don't you, that if you keep allowing the alt-right (or anyone else) to define what terms are racist, you give them complete control of what language you can use, right?
[ link to this | view in chronology ]
Some terms, the alt-right can keep. That’s one of them.
[ link to this | view in chronology ]
and they thought the IT budget was just fluff
[ link to this | view in chronology ]
Re:
Well considering how little the IT guy apparently did, it arguably was.
[ link to this | view in chronology ]
IT for a municipality is VERY difficult.
I've work as IT for a small municipality (for a short period). The job is very difficult.
There is minimal budget; trying to get money for backup hardware, storage, time to prepare and test restoration plans is hard (the word impossible is more realistic).
The situation is worse if one or more public officials doesn't support the effort. Many times municipal IT is merely a political football, being kicked around in the internal political battles.
All it takes is one official who uses IT as a weapon, who believes that because said official can (barely) do email and create a document, that the said official knows more than IT.
Part of the villainy is Microsoft. The pretense that Microsoft will take care of it, and maybe some "cloud" backup is all that is necessary has ignorant officials thinking that they are safe at little cost.
Without the full facts about the performance of the Baltimore's IT boss, Mr. Johnson, we can't really make an informed judgment. It is possible that Frank Johnson didn't do the job he was paid to do. It is also possible that Mr. Johnson did the best he could under difficult and even hostile conditions. In either case the senior officials of Baltimore are to blame. They should have either replaced Mr. Johnson or supported him. Further, the citizens of Baltimore are to blame, they supported the two party status-quo which promise them everything and throw crumbs.
Further, we are to blame. We sit here complaining. How many of us have fought local, state and the federal government in order to obtain either proper funding for IT, or a mandate to minimize the collection and storage of data. How big an effort have any/all the readers here performed to have a rational data PURGE policy for government?
So, having been there, and done that and left when I knew that no matter how good or poor I was at IT, I couldn't make city hall understand or accept objective reality. However, at least I tried.
AC
[ link to this | view in chronology ]
Business As Usual
My experience in large organizations is that department heads and other higher level, higher paying jobs are allocated by office politics and favoritism (cronyism?) not competence. IT especially is vulnerable to this, since far more competent people in the field are very poor at skills needed to gain promotions. People tend to understand, let's say, accounting better than IT and the Accounting boss needs a certain level of certification.
Looks like this guy got a job by being the schmoozer, and then coasted; meanwhile earning brownie points from above by avoiding necessary upgrades in order to demonstrate budget control.
Like Brownie during Katrina, the relative competence and preparedness is then demonstrated during a crisis that was or was not properly planned for.
[ link to this | view in chronology ]
Re: Business As Usual
This is so true. I've working in IT for 25 years and have found that the overwhelming number of failures like this are caused by piss-poor management. From supervisors all the way to the C-level. The IT guys in the trenches get the blame, but they're usually the ones trying to do the right thing while having to fight against said piss-poor management or lack of budget.
[ link to this | view in chronology ]
Re: Re: Business As Usual
Not to say there isn't good management in IT, but those folks usually don't last long. They either quit in frustration, or get let go by idiots above them because they didn't toe the party line.
[ link to this | view in chronology ]
Re: Business As Usual
Yeah, I wouldn't be surprised to learn that whoever had the job before Johnson had pushed for redundancy etc., with reasonable price tags, and was pushed out and replaced by someone who could bring the department in under budget.
[ link to this | view in chronology ]
Often desktops will have options for data storage, local and some sort of network file. The second is backed up. The thing is, using the network files isn't intuitive to most people's mental workflow, so files go right on the desktop, like they do at home, which may not be backed up. Arguably a training issue, and maybe there could have been better options for document storage as well.
I'm not sure if this happened here or not but I can also just about guarantee you that the IT department was underfunded and staffed. The question then becomes why?
The why isn't very tricky, no one sees the network, no one funds the kind of practices and security, and, we / us, whine about overpaid do-nothing government workers. I'm not saying the IT leadership wasn't awful, it can be, and the guy talking about politics has it right, but anytime you see the IT director getting hammered, double-check where the money went, how much of it there is, and make damn sure the city manager and council gets asked hard questions about their involvement in things.
You might be surprised at what got funded over IT and government is, never, simple.
[ link to this | view in chronology ]
From the Ars Technica article; "Johnson, who was hired by the now-resigned Mayor Catherine Pugh, was previously a regional vice president of sales for Intel with no IT operations experience."
[ link to this | view in chronology ]
Back-up
Even on my humble home network of six computers, every PC has several external drives and everything is all on there, duplicated all around the house! Might be a touch of over-kill but that's the way I like it. I do have other anti-ransomware safeguards as well. With over 500 gigs of music, video and docs. on the system, I would not exactly be best pleased if anything happened to that little lot. How come corporations with IT departments and loads-a cash seem unable to engineer reliable back-ups? Is it I.T. versus the bean counters?
[ link to this | view in chronology ]
“That can’t be right? That’s real?” Costello asked.
I'm waiting for the punchline, what did Abbott say?
[ link to this | view in chronology ]
Generally poorly staffed govt IT.
Given the relatively low pay (compared to business/corporate work, usually < 75% of the pay) and the high friction (incredibly lengthy interview/background processes) to get into a government job. It doesn't come close to optimizing for skilled, competent, go-getter types at all.
The above is why I generally (some military and police exceptions) don't trust people who worked in government for more than a couple years in terms of hiring/interviews. They're horrible systems to work in at times and it's often better to work for adjacent consulting companies in the space than the agencies. Not that it's always the case, some states/counties/cities are better than others.
In the end, it doesn't surprise me at all. Generally I don't consider something backed up unless it's on 3 different mediums/devices in at least two distinct geographic locations. Beyond this, it's better to drop to a relatively secure location, and have your backup infrastructure pull from that drop location into the backup system. This is a better separation than push, which a compromise like this could effect backups as well.
Disclaimer, I do work in a company that provides services/hardware/software for govt work. My opinions are my own and do not reflect the company I work for.
[ link to this | view in chronology ]