The City Of Baltimore Blew Off A $76,000 Ransomware Demand Only To Find Out A Bunch Of Its Data Had Never Been Backed Up

from the nice-work-if-you-can-get-paid-to-do-it-and-then-not-do-it dept

The City of Baltimore was hit with a ransomware attack in May of this year. Criminals using remodeled and rebranded NSA exploits (EternalBlue) knocked out a “majority” of the city’s servers and crippled many of its applications. More details didn’t surface until September when the city’s government began reshuffling the budget to cover the expenses of recovering from the attack.

The person in charge of the city’s systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn’t done much IT directing.

Johnson, who also serves as the city’s chief digital officer, received significant criticism from local authorities for the response to the May 7 attack. City council members alleged a lack of transparency and communication in the wake of the incident, as well as an inability to maintain a functional organization “during an emergency event.” He also also never drafted a continuity of operations plan for an IT attack of the kind that occurred.

It looks like the list of stuff Johnson was being paid to do that he never did. Hence the catastrophic outcome when the city refused to pay the $76,000 ransom. Given the fact that $6 million has already been pulled from parks and public utilities funds to “harden” city systems, the $76,000 demand now seems like a bargain.

City residents should be asking WTF their tax dollars are being spent on. The city’s audit of its compromised system rolls on, delivering even more embarrassing details about the city’s IT skill set. (via Ars Technica)

A new audit of Baltimore’s information technology department says the agency lost key data during May’s ransomware attack because some in the agency used an outdated method for storing files: the hard drives on their individual computers.

[…]

“Performance measures data were saved electronically in responsible personnel’s hard drives,” [Baltimore City Auditor Josh] Pasch reported. “One of the responsible personnel’s hard drive was confiscated and the other responsible personnel’s selected files were removed due to the May 2019 ransomware incident.”

Bureaucracies are prone to understatement and the assessment of the ghastly state of affairs by Pasch was no exception. According to Pasch, the permanently-missing data resulted in a “loss of confidence” in the city’s IT department’s ability to do its job.

This understatement brought the hearing to a halt as council members expressed their disbelief that city data was not being backed up. Their comments were less understated.

Hearing that, City Councilman Eric T. Costello, a former government IT auditor himself, stopped the hearing.

“That can’t be right? That’s real?” Costello asked.

It’s apparently real. City data needed for an audit cannot be recovered because the IT department never made an effort to express the dangers of storing the only copy of data locally. It also apparently never made a push to create cloud backups of important files. When the ransomware struck, the stuff locked up was — in far too many cases — to only copy of that stuff.

The tragically hilarious postscript to this is the city’s response to Ars Technica’s request for info on the city’s cyberattack recovery plans.

Ars has requested information from the city regarding the contracting details for the recovery, but the city has thus far provided no data. Requests for data on the status of patches and disaster recovery plans were refused because the documents do not exist as a result of the ransomware attack.

It’s easy to mock governments for their inability to properly handle the massive amounts of data they collect, create, and retain. And so we shall. The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you’re going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec.

Filed Under: , , ,

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “The City Of Baltimore Blew Off A $76,000 Ransomware Demand Only To Find Out A Bunch Of Its Data Had Never Been Backed Up”

Subscribe: RSS Leave a comment
44 Comments
This comment has been deemed insightful by the community.
That One Guy (profile) says:

'Always back up your stuff': A good idea no matter the scale

I see the idea that everyone needs to learn to back up their stuff the hard way applies to government agencies as well as individuals. Just a pity that people keep having to learn that lesson the hard way, rather than learning from those that came before.

R2_v2.0 (profile) says:

Re: Re: Re: Re:

LinkedIn completely checks out:
Leads all digital transformation programs and supports the Mayor’s ongoing efforts to modernize the City of Baltimore’s IT capabilities, which also include scaling the local IT ecosystem to drive awareness & tech investment in Baltimore City.

Now the city has to modernise and invest. Job done.

Anonymous Coward says:

Re: Re: Re:

It takes far more than a few hard drives and some FOSS software to back up all of the data generated by every employee of every agency for a city the size of Baltimore. The data storage costs alone would be massive, and constantly growing.

Some other asshole probably thought it should only take very little effort and resources too, so that’s what IT got instead of a proper budget and staff, and that’s how the city ended up where they are.

A Guy says:

Re: Re: Re: Re:

Once again, I said competent IT guy. Most of the information could likely be reduced to text and backed up with a greatly reduced memory footprint.

Water bills, power bills, typed police/fire/ems reports ect.

Some information may not be able to have their text extracted and reduced like audio and visual recordings. You only need to save image producing files (i.e. there website formatting or fire report formatting) once also. They don’t run youtube or something. The fact is hard drives are huge now and a few of the large ones could likely backup the city of Baltimore.

You should probably not apply for IT positions.

A Guy says:

Re: Re: Re:3 Re:

What is your specific problem? If I was doing a cheap semi-lazy (but competent I hope) job of it I wouldn’t bother writing scripts to extract the text and then having the document reconstituted into the proper file structure when the data had to be restored. That would lead to really compressible data but it might be a much larger workload for programmers than you have budget for.

A cheaper and easier solution that’s much lighter on programming is to get a backup solution that downloads required files from pre-specified folders on a schedule (monthly/weekly whatever). Obviously it is a good idea to stagger backup schedules so nothing gets overloaded in the process.

Finding and running the best compression algorithm for the expected data-type would be my next step. Here is a quick set of replies for the question "what is the best compression algorithm?"

https://www.quora.com/What-are-data-compression-algorithms?redirected_qid=18888451#

As some of the answers note there is no one best compression algorithm for every file so some research and experimentation to find good algorithms for the file-types you are backing up may be necessary.

You would have to get the city departments to cooperate by dropping their work product into the right specified folders and you may have to investigate their computers to find the correct folders to backup too but that can easily be made part of the job. It’s not too difficult though it may be time-consuming for a large city.

FOSS solutions on the market already have the ability to just download a folder instead of imaging an entire drive. You can also make them just download specified file types in a folder if that is more optimal.

I would also create 2 backup server/computers that run on alternative schedules so if one set of backups is corrupted you have a good chance of recovering from the other, even if is slightly older. You would have a better chance of being able to recover most of the data.

These are just general thoughts on the issue. If you have a specific problem I would give you my "I hope somewhat competent" opinion if you desire.

This comment has been deemed insightful by the community.
Gary (profile) says:

Targets...

The fact that so many municipalities have paid out ransoms recently has painted a huge target on every hospital, town, and county government. They made the right call in not paying the ransom!
Some insurance companies are just "Insuring" by paying the ransom, causing the problem to spiral downward.

Now, not backing up. (Or backing up on TAPE), that is the wrong decision.

Anonymous Coward says:

What I don’t understand is this: I have backed up my entire home network since the 90s, and use a dual NAS solution plus rotating offsite backups for over a decade. This entire solution cost me a total of around $400 over 10 years ago, plus remembering to take the offsite drive with me when I go to the bank. And I’ve got redundant backups of all data on all devices on my network.

Surely an IT department can manage at LEAST this level of data security for their endpoints?

Anonymous Coward says:

Re: Re:

Back when I worked in IT, I created a backup solution that merged and deduplicated all files from all endpoints on my networks, and it included a web interface where individuals could use their AD logins to recover any files backed up on endpoints that were under their management. The backup repository could be stored anywhere, as it was encrypted. The software to do so was free; it just took me a few hours to configure and add to local policy for administrative file access.

Essentially, this will work for any device on a network that allows network access. It also would have been resistant to a ransomware attack, as it was a one-way system. New writes just get deduplicated; deletions required direct access, which means admin account credentials to a locked down system.

This comment has been deemed insightful by the community.
Anonymous Coward says:

I'm sorry, but I must respectfully disagree.

Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain.

Some portion of that $6 million (or the estimated $18 million) may be going to data recovery: pulling things off backups, rebuilding data from hardcopy (dead tree data), or going back to original sources and polling institutional memory when no other method works.

But the rest of it, to "harden" city systems? That’s pure technical debt. Money that should have been spent and hadn’t been. Systems that are not vulnerable to the latest exploit don’t come about by themselves, they get maintained – patched, backed up, put on new hardware when necessary. Security processes get evaluated and updated. Backups get made. Worst case scenarios get gamed out.

If you wish, consider the ‘rebuilding’ cost as the cost of a hard security audit. And that cost, as a whole, is likely more expensive than if they’d put a security audit into the budget in the first place.

This comment has been flagged by the community. Click here to show it.

This comment has been flagged by the community. Click here to show it.

anonymous coward says:

IT for a municipality is VERY difficult.

I’ve work as IT for a small municipality (for a short period). The job is very difficult.

There is minimal budget; trying to get money for backup hardware, storage, time to prepare and test restoration plans is hard (the word impossible is more realistic).

The situation is worse if one or more public officials doesn’t support the effort. Many times municipal IT is merely a political football, being kicked around in the internal political battles.

All it takes is one official who uses IT as a weapon, who believes that because said official can (barely) do email and create a document, that the said official knows more than IT.

Part of the villainy is Microsoft. The pretense that Microsoft will take care of it, and maybe some "cloud" backup is all that is necessary has ignorant officials thinking that they are safe at little cost.

Without the full facts about the performance of the Baltimore’s IT boss, Mr. Johnson, we can’t really make an informed judgment. It is possible that Frank Johnson didn’t do the job he was paid to do. It is also possible that Mr. Johnson did the best he could under difficult and even hostile conditions. In either case the senior officials of Baltimore are to blame. They should have either replaced Mr. Johnson or supported him. Further, the citizens of Baltimore are to blame, they supported the two party status-quo which promise them everything and throw crumbs.

Further, we are to blame. We sit here complaining. How many of us have fought local, state and the federal government in order to obtain either proper funding for IT, or a mandate to minimize the collection and storage of data. How big an effort have any/all the readers here performed to have a rational data PURGE policy for government?

So, having been there, and done that and left when I knew that no matter how good or poor I was at IT, I couldn’t make city hall understand or accept objective reality. However, at least I tried.

AC

Anon says:

Business As Usual

My experience in large organizations is that department heads and other higher level, higher paying jobs are allocated by office politics and favoritism (cronyism?) not competence. IT especially is vulnerable to this, since far more competent people in the field are very poor at skills needed to gain promotions. People tend to understand, let’s say, accounting better than IT and the Accounting boss needs a certain level of certification.
Looks like this guy got a job by being the schmoozer, and then coasted; meanwhile earning brownie points from above by avoiding necessary upgrades in order to demonstrate budget control.
Like Brownie during Katrina, the relative competence and preparedness is then demonstrated during a crisis that was or was not properly planned for.

lucidrenegade (profile) says:

Re: Business As Usual

This is so true. I’ve working in IT for 25 years and have found that the overwhelming number of failures like this are caused by piss-poor management. From supervisors all the way to the C-level. The IT guys in the trenches get the blame, but they’re usually the ones trying to do the right thing while having to fight against said piss-poor management or lack of budget.

Anonymous Coward says:

Often desktops will have options for data storage, local and some sort of network file. The second is backed up. The thing is, using the network files isn’t intuitive to most people’s mental workflow, so files go right on the desktop, like they do at home, which may not be backed up. Arguably a training issue, and maybe there could have been better options for document storage as well.

I’m not sure if this happened here or not but I can also just about guarantee you that the IT department was underfunded and staffed. The question then becomes why?

The why isn’t very tricky, no one sees the network, no one funds the kind of practices and security, and, we / us, whine about overpaid do-nothing government workers. I’m not saying the IT leadership wasn’t awful, it can be, and the guy talking about politics has it right, but anytime you see the IT director getting hammered, double-check where the money went, how much of it there is, and make damn sure the city manager and council gets asked hard questions about their involvement in things.

You might be surprised at what got funded over IT and government is, never, simple.

Dave P. says:

Back-up

Even on my humble home network of six computers, every PC has several external drives and everything is all on there, duplicated all around the house! Might be a touch of over-kill but that’s the way I like it. I do have other anti-ransomware safeguards as well. With over 500 gigs of music, video and docs. on the system, I would not exactly be best pleased if anything happened to that little lot. How come corporations with IT departments and loads-a cash seem unable to engineer reliable back-ups? Is it I.T. versus the bean counters?

tracker1 (profile) says:

Generally poorly staffed govt IT.

Given the relatively low pay (compared to business/corporate work, usually < 75% of the pay) and the high friction (incredibly lengthy interview/background processes) to get into a government job. It doesn’t come close to optimizing for skilled, competent, go-getter types at all.

The above is why I generally (some military and police exceptions) don’t trust people who worked in government for more than a couple years in terms of hiring/interviews. They’re horrible systems to work in at times and it’s often better to work for adjacent consulting companies in the space than the agencies. Not that it’s always the case, some states/counties/cities are better than others.

In the end, it doesn’t surprise me at all. Generally I don’t consider something backed up unless it’s on 3 different mediums/devices in at least two distinct geographic locations. Beyond this, it’s better to drop to a relatively secure location, and have your backup infrastructure pull from that drop location into the backup system. This is a better separation than push, which a compromise like this could effect backups as well.

Disclaimer, I do work in a company that provides services/hardware/software for govt work. My opinions are my own and do not reflect the company I work for.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Ctrl-Alt-Speech

A weekly news podcast from
Mike Masnick & Ben Whitelaw

Subscribe now to Ctrl-Alt-Speech »
Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...