One year ago – almost to the day – the city of New Orleans was the victim of a cyberattack and following the city’s impressive recovery, MeriTalk checked in with New Orleans CIO Kimberly LaGrue to understand how her team helped the city recover and what they’ve learned as a result.
In the early morning hours of Dec. 13, 2019, cybercriminals began to attack the city government’s networks. By 11 a.m. that same day the attack was detected and LaGrue and her team sprung into action. The recovery process lasted for months and was interrupted by the COVID-19 pandemic. Twelve months later, LaGrue is proud of how her team – along with private sector partners – was not only able to recover from the attack, but use it as a launching pad for further modernization.
The interview below has been edited for clarity and brevity.
MeriTalk: Can you walk us through your team’s thought process in those first days and hours after the attack was discovered?
LaGrue: We were really clued in to these things because of what was happening at the state level. So for us, this was a spring into the action that we had rehearsed. We knew we wanted to immediately confer with experts who had dealt with similar attacks in the state and throughout our area and get their recommendations.
Based on the recommendation from the state and our security partners, we undertook a full re-implementation of our environment – we wiped all of our computers and we sanitized all data in servers. With the recommendation that we take no chances and fully clean our environment to avoid reinfection, we had to get working right away. We wanted to establish a game plan that allowed us to reach all 4,000 of our users, all 3,000 of our workstations, and over 100 locations as quickly as we could. We established some effective communication protocols at the very beginning and tried to get information out to our users as quickly as we could, and dispatch help to them as quickly as we could.
MeriTalk: Obviously your team worked hard to help New Orleans recover, but we know that due to limited budgets and staff, local governments do lean on private sector partners, especially during a crisis. How has your team worked with private sector partners, like Pure Storage, in the recovery process?
LaGrue: We worked with them early. We knew very fundamental things had to be done – we needed a clean environment, we needed clean data, and we needed clean platforms. The first things we did were efforts focused around having a clean place to land – whether it was a computer, a server, storage, or networking – everything had to be fresh. Our partners really came in and helped us figure out what a sanitized environment would look like, and how we would build it. Very early on, partners like Pure Storage understood that to clean and sanitize the level of data that we had, we had to have new platforms to move that data to and that we would want high performing platforms to be able to store our information whether it was for deep inspection, or just to get up and working as quickly as possible.
MeriTalk: We hear frequently that when it comes to cyberattacks the question is “when,” not “if” in terms of whether a government will be a victim of an attack. That aside, do you have any advice for governments of all levels, but especially major cities like New Orleans, that are doing their best to avoid an attack?
LaGrue: Be as ready as possible, because it is “when,” not “if.” Those attacks can be of great magnitude or have a very small impact on your environment. But knowing what you will do at every turn is most important. And it is the thing that you can actually do without a lot of resources or without a lot of outside help. You can prepare, you can be prepared, and you can have a plan. Just like a fire drill – knowing what exit this part of the staff is going to use and what exit that part of the staff is going to use and rehearsing those things is really important. Also, understanding what your environment looks like – this is the time to inspect the environment to know what you have and to order or rank the scenes in order of priority that have your assets, whether it’s servers or applications, and how you would bring those things back up.
We have for years gone through exercises of what we call the “bus through the window” scenario. So, we always wanted to think about what scenarios could impact or cripple our operations. Because we live in Louisiana and in the heart of a gulf bay area, we have natural disasters that plague us as much as the cyber event could. Our scenarios were pretty thorough, and I think whatever things are relevant in the area, that could physically affect an environment, as well as things that could have a cyber effect on the environment should also be considered.
A good disaster recovery plan should be an iterative document. It is something that we work on pretty often. A disaster recovery plan is really important, and then having a backup to your backup is great as well. What we realized early on was is that we couldn’t get to any electronic systems, we just needed one paper copy of our disaster recovery plan. We had on our walls visual graphics of our network infrastructure, so we did not have to rely on electronic systems for that. Those things sound like simple things, but they are part of an invaluable plan for recovering from any type of disaster, and having built those things over the years was important to us. My recommendation is if you have not yet, just get started and do the things that you can do immediately. Just know what you have and understand how you would react if some event presented itself. Just have a plan.
MeriTalk: As with most governments, we’re sure New Orleans has broader IT modernization plans. How has the cyberattack impacted broader goals your team has?
LaGrue: It has accelerated our modernization plans. We did modernize some platforms. It made us look at some legacy applications and tools, and it accelerated their replacement. Although we were acting quickly, we took a little time to really analyze the cost of standing up physical infrastructure in our environment and where it made more sense to migrate to the cloud. For instance, our email and business application platforms are cloud-based now, but we saw that we had an opportunity to increase our storage footprint and where we really needed to keep things on-premise and look for a solution that was more affordable, that would serve us better in the long run, and give us the growth that we needed.
That’s where Pure Storage came in, we were able to leverage that platform to help us do the things that we knew we would need to do in the short and long term for our departments in terms of the way we consume and manage data.
I think everyone’s appreciated and benefited from what it took for us to fully recover from an attack of this magnitude. Starting from scratch is just a lot easier than having to wonder if you got it all and if everything was really clean. It was an extensive project but the peace of mind that we have and the security that we have has been well worth it.
