Phoenix Protocol: Rising from the Ashes After a Cyberattack

Traditional disaster recovery plans assume natural events with no malicious activity. Cyber attacks involve active adversaries and hidden infections, requiring cautious, methodical recovery to avoid reintroducing threats. 

The Phoenix Protocol is Commvault’s post-attack recovery framework built on four pillars: AirGap Protect, an Isolated Recovery Environment, a Cyber Recovery Plan, and continuous chaos testing to strengthen readiness. 

Many companies do not plan or practice recovery until an attack occurs. Unprepared teams often take independent actions—such as restoring systems prematurely—that can spread malware and worsen the incident. 

Chaos testing simulates unpredictable, high-pressure recovery scenarios, allowing organizations to validate processes, uncover vulnerabilities, and strengthen team coordination before a real incident. 

AirGap Protect prevents tampering by separating critical backups from production systems, while an Isolated Recovery Environment allows teams to safely analyze, test, and restore clean systems without risking reinfection. 

Please view video here for a time-stamped transcript

Welcome to today’s SHIFT podcast session, the Phoenix Protocol: How to Rise from the Ashes
after a Cyber Attack. 

I’m Chris Dirado and with me today is Michael Stempf. 

Michael, thanks for joining me today, I’m really looking forward to our conversation. 

Thanks Chris. 

When a cyber attack hits recovery can seem overwhelming, but it’s also a moment to rebuild
stronger. 

In this conversation, we’ll explore the Phoenix Protocol and how our four pillars here at
Commvault can help you rise from the ashes. 

AirGap Protect, Isolated Recovery Environment, a Cyber Recovery Plan, and most
importantly, testing for chaos. 

Together, we’ll look at how organizations can transform from this crisis into an
opportunity for renewal and how to come out from an attack on top. 

So, Michael, what are some of the most common mistakes organizations make when recovering
from a cyber attack? 

So that’s a trick question. 

The most common problem people have in the recovery is they don’t plan for it until the
recovery has to happen. 

You gotta be prepared, right? 

The big event, we like to call it bang, and people think about what happens right after
the event’s occurred. 

But if you haven’t planned, if you haven’t practiced, if you haven’t built up that mental
strain that you’ve gone through it over and over again and worked with your teams, 

to build a team building experience around it, you’re not gonna be prepared. 

After the fact, what we’ve seen a lot of times is people honestly trying to do their best. 

You’ll have a Windows admin who just wants to help and get their servers back up and
running, and so they’ll go off kind of rogue and do shadow IT and bring up their own 

systems. 

And we’ve seen time and time again when they’re doing these sort of things, they just
reinfect a new area that the bad actor wasn’t in, 

and then you have to go forward and deal with that again. 

So you perpetuate the problem by trying to do the best job. 

And just without that practice, without testing with the chaos that you talked about,
they’re not going to understand the really level of intricacy and advancement that these 

bad actors bring against us. 

And you bring up a really good point there about people just trying to help, but
sometimes… 

you know, getting ahead of themselves, right? 

Initiating recoveries before we’re actually ready to recover. 

And I know from my experience and yours too, we’ve been in a lot of these recovery
scenarios. 

You sometimes you’re working 60, 70, 80 hour work weeks. 

The worst time to do something that doesn’t have a good end result is while you’re already
running razor thin and full throttle, right? 

So for my next question, I want to talk about some of the fallacies that are… 

you know, that we think we’re protected, but really, you know, these are these are just
fallacies and we’re not really they’re not really helping us in the end. 

Could you give me some examples of some of those fallacies? 

Yeah, I think the first and foremost is everyone has a disaster recovery plan. 

They’ve had them for years. 

They’ve tested them for years. 

And cyber is just another disaster or so they believe. 

The problem with that is disaster recovery is all about speeds and feeds. 

Right. 

It’s a natural disaster, 

tornado, hurricane, whatever it might be. 

And there’s no malicious intent. 

And when you get into cyber, speeds and feeds just simply mean that you’re gonna reinfect
the environment. 

You’ve gotta go slow, you’ve gotta be methodical, you’ve gotta think about that malice,
the bad actor being in there. 

And so utilizing a plan that was specifically designed to get you back up and running as
fast as possible, it’s just gonna cause issues. 

So you have to slow down, take a breath. 

You have to have a cyber recovery plan, which is maybe at its core a lot of what’s in the
disaster recovery, but the ultimate goal is how do I come back clean, not how do I come 

back as fast as possible. 

So that’s really the primary one. 

But there’s so many that go along with that. 

You know, first and foremost, I’ve actually seen just about everything destroyed in a
cyber attack that you could. 

And one of the precursors to that is that everybody has 

everything tied into an identity management system today, say Active Directory. 

Active Directory latest statistics is it’s in 98 % of all enterprises and it’s attacked in
92 % of all cyber attacks. 

Of course, right? 

You can go anywhere once you’re in Active Directory. 

It’s the keys to the kingdom. 

So one of the things that I always do recommend is uh pull your last line of defense, pull
your data protection system out of Active Directory, 

because one of the common things bad actors do, because they know if your data protection
survives the attack, there’s odds are that you’re not going to pay. 

That’s how you get out of paying. 

Absolutely. 

One of the things that we say is pull out your data protection from Active Directory, make
it standalone, make it separate, make it isolated as much as possible. 

So as much of your enterprise that gets destroyed, that is still going to survive. 

So quick note, Michael brought up a good 

a good note there about a cyber recovery plan. 

In some of the sessions that I’ve done, a lot of folks I found did not have cyber recovery
plans. 

Well, at Commvault, we have you covered. 

If you go on our readiness platform and search for a cyber recovery plan, you can go ahead
and you can download our template and you can make it your own. 

And as Michael said, some of the things will be in your DR plan. 

Go ahead and borrow those elements, right? 

Copy and paste. 

But having a cyber recovery plan is really what’s going to help bail you out once you’re
in one of these scenarios. 

Michael, how have recovery expectations changed from just getting data back to getting our
business back? 

Good question. 

So, you know, most people plan for one Z, two Z type restores. 

I have a couple of files here. 

I have a server here. 

And you can plan for that, right? 

It’s it’s very mathematical, right? 

For a D.R. 

I’ve got a floor. 

I’ve got a server. 

I have a building. 

I have to know how much data is there, how much pipe I have to the next 

location, geographically dispersed location, and I can run scenarios all day long on that. 

The problem with a cyber event is I don’t know the extent, right? 

How many servers did they hit in this organization? 

How many locations did hit? 

I had a customer who was hit. 

They had 99 locations worldwide, and the bad actor destroyed 99 locations worldwide. 

So every copy of data they had anywhere was absolutely destroyed. 

Looking at it and designing this and running through and practicing it is really some of
the most important things here so you’ve got to make sure that you treat it differently 

that you look at it differently and The outcomes are always going to be different. 

And you bring up a really good point there. 

There are three distinct different kinds of recovery operations we plan for right. 

Operational recovery, disaster recovery, and cyber recovery. 

Can you explain a little bit more on each of those and how they’re different and and how
that 

is really preventing customers from not being able to recover in a cyber event? 

Sure. 

Operational recovery, that is the typical, I deleted a file, you have a local copy that
you can restore from, super fast, easy to do, automated in most places, you can simply go 

in and… 

I was gonna say, here at Commvault, we all have the ability to do our own recoveries on
our laptops. 

I’m sure most organizations, it’s simple, right? 

Simple procedure. 

Absolutely. 

The disaster recovery is, for those natural disasters, 

and that’s very advanced. 

We’ve been doing that for over 30 years, testing it quarterly in most places. 

In fact, we did it so much people got lazy with DR. 

They’re not really testing anymore. 

What they typically do now is they flip-flop their data center from one site to another
every six months and say, hey, we’re running in production. 

It’s a good test. 

But then with that last one, that with cyber, it requires some specific things. 

One, and I think the most important, is a tertiary copy of data. 

A third copy of data that’s behind somebody else’s infrastructure because, you know, the
bad actors are going to have access to AD, they’re going to have your entire environment, 

they’re going to have all of your cloud credentials. 

So having just that second copy of data that you would normally geographically disperse
that, it’s not enough protection. 

So you have to make sure this is behind somebody else’s infrastructure, that it is
immutable and indelible. 

Indelible puts that governance on it to make sure that 

no one, not even with elevated credentials that they could have gotten from AD, can change
or delete that data. 

That’s the data that’s gonna bail us out once we’re under attack. 

We know it’s gonna be there in the time, because it can’t be deleted like you said, it
can’t be changed, it has to meet strict requirements. 

And then for a bad actor to get a hold of it, they wouldn’t only have to breach your
organization, they’d have to breach our own, right? 

So it’s really a two-pronged attack there that they would have to get to. 

So with that… 

we know that we can have a safe, secure copy of data. 

Where should we restore that data to in the event of a cyber attack? 

That’s the point nobody ever thinks about, right? 

Roughly 17 % of all attacks are destructive attacks, where they get into the firmware and
the bios of the servers, and you can’t recover to them. 

But that’s in everyone’s plan. 

I’m going to go back to exactly where it came from. 

And nowadays, even though that 17 % is a fairly small number, most CISOs 

because you don’t know if it was a destructive attack, they’re like, I have to assume it’s
destructive. 

So most companies don’t have a plan where they have 50 servers sitting out on their dock,
ready to go. 

And so in today’s environment, you must have an isolated recovery environment. 

One, it’s a safe place that I can recover to, and two, it’s a place that I can promote
into production and that I know the bad actor’s not in. 

So as one of those key areas that you talked about earlier, 

an isolated recovery environment, we call ours clean room, is essential for cyber
resiliency. 

Sure, so now we have a good, we know we have copies of our data, safe and secure, and now
we have a place where we can now land that data, and to your point earlier, we can clean 

and vent that data before we move it into production. 

And one of the features I think here at Commvault that doesn’t get enough hype is our
any-to-any portability, right? 

Because as you said, you may have a destructive on-prem attack, well now you need to
pivot, right? 

You need to… 

you need to go somewhere else, how long is it to get new hardware? 

Six weeks maybe? 

You can’t sit around six weeks without production servers. 

You can pivot your workloads to your cloud vendor of choice or a different hypervisor, and
it really gives you the flexibility to recover after one of these attacks. 

That’s actually one of the key areas that’s never really talked about that Commvault has
over its competitors is the fact that we’ve been doing any to any for years. 

So if you wanted to… 

have an on-prem VMware environment and then move that to AWS EC2 environment, it’s all
done automagically. 

Happens on the restore, I don’t have to worry anything about it. 

And the table of the any to any different variables that we have from what platform to
other is absolutely incredible and just totally eclipses anyone else in the industry. 

Yeah, we really give our customers all the options they could ever want, right, when it
comes to that? 

Absolutely. 

All right, so let’s pivot here. 

What kind of challenges are you seeing most frequently right now with these cyber attacks? 

The number one thing is the lack of preparation. 

It’s dealing with it as a disaster. 

When you talk about a disaster recovery and recovering from that, you would typically have
a backup admin, right? 

And this person would, in that case, 

I always call them, they’re considered to be a god, right? 

When you get out to the disaster recovery side, it’s like, don’t go near this person,
don’t get near them, just hand them food and drinks every so often, and just let them go, 

because they’ve worked out the plan forever. 

And it was all their responsibility. 

The problem is, is that person almost has no responsibility in a cyber attack. 

They can’t just start recovering things because they’ll start a reinfection phase. 

You’ve gotta work with your security teams, your IT teams, your legal, your comms teams,
all of these people have to work together and they’ve never worked together before. 

And I always like to joke that probably the last time your security team and your IT team
got together is when they played against each other in some baseball game at a corporate 

event. 

They’re always against each other, never on the same team. 

But when you get into a cyber event, if I’m the backup person, I… 

I don’t have authority to do a recovery until somebody from the security side tells me to
do it. 

And the security people can’t do what they need to do until somebody from the IT side has
to do it. 

So it’s this idea of checks and balances, which is a totally foreign concept in any data
recovery before cyber events. 

Yeah, and you know, what’s funny to me is that, think about the last time you interacted
with a backup admin, right? 

It was probably, hey, I lost an email or a spreadsheet, and what’d they do? 

They restored it in minutes, right? 

Now, to your point, they can’t. 

They can’t go doing that because you don’t know what’s been poisoned. 

You don’t know if they’ve hit a dozen servers, if they’ve hit the entire ESX farm, and you
can’t just start restoring stuff and that’ll just put you back at right where you started. 

It really needs to be a team effort, right? 

Everyone finally has to come together, work together, play ball to get the organization
back up and running. 

Agreed. 

So what are some of the biggest lessons organizations learned after they’ve gone through a
cyber attack? 

Lessons learned vary. 

They’re all over the place. 

I think the most important thing is, and it typically comes after the second attack, which
it’s very common to see up to four attacks or so a year, is that it’s not truly planable. 

You’re not going to be able to set up scenarios that are going to pan out exactly as you
see it when that cyber event happens. 

There’s so many different variables. 

Here’s the thing, in disaster recovery, it’s very easy to plan for a tornado. 

A tornado is probably going to take out a building. 

A flood can take out… 

maybe lower levels of a building, but it’s planable, it’s rehearsable, I can go in and do
it. 

With cyber, there are so many different ways that they can attack. 

So many different ways they can get in, so many ways they can destroy information. 

Heck, nowadays, it’s not even so much about all ransomware, right? 

We have attacks happening just because somebody wants to take out a competitor, or one
government wants to take out another government. 

That’s just pure malicious intent with no payback on the back end, other than destroying
your enemy. 

That’s hard to plan for. 

Sure. 

And so what we have to do is really up that chaos theory when we do our testing. 

One of the common things that I always like to do is whenever I go into these tests, they
always have a predefined list of what was attacked, right? 

They always know exactly what servers and what’s nice is they always have everybody who’s
responsible for those servers in the meetings with us. 

Absolutely. 

And one of things I always like to do is I like to write the names of servers down in the
back of playing cards, 

and I’ll throw those cards against the wall and whichever ones land face up, those are the
ones that just got hit. 

So that is a great way of bringing that chaos methodology in here so you can test. 

And once you’ve tested in so many different chaotic ways, I’m not gonna say that you’re
prepared for that attack, but you have an idea and understanding of what the impact’s 

gonna be from it and then how you can recover better. 

Yeah, and you know your pivot point, right? 

Hey, if I know if X happens, I’m gonna pivot to Y, right? 

If I know if I do Y and it doesn’t work, I’m gonna pivot to Z. 

And I think you can’t be prepared for all of these scenarios, but the more of them you’re
prepared for when this actually does happen, it’s that muscle memory, right? 

Like, hey, I know that we’re gonna be able to do this because I prepared for it, I’ve
prepared for all these situations. 

I know that the FBI a few years ago used to say, hey, if you get hit with a cyber attack,
you need to do these set of procedures. 

And then a year later, they changed it to, 

when you get hit with a cyber attack, you need to do this. 

And now this year they say, for the amount of times you’re gonna get hit with a cyber
attack, you need to be ready, you need to be prepared. 

Can you talk a little bit more about what preparedness looks like there? 

Yes, so it is frequently, and so I think the number one thing in preparedness is to change
your mindset. 

And the mindset changes not if or when or how frequently, but it’s really, I have to
assume that I’ve already been breached. 

You know, we always like to joke that there’s two types of companies in this world. 

Those that know they’ve been breached and those that don’t know they’ve been breached. 

And while that’s not entirely true, I mean, let’s be honest, last year in America, it was
only 60 % of companies that had a breach, right? 

So you had a chance, you had a 40 % chance that you weren’t breached, but that was one
year. 

Yes, lucky. 

But if you take the mindset that I’m assuming breach, 

when I go into work tomorrow, I’m gonna make different decisions. 

I’m gonna look at things differently, I’m gonna analyze things differently, and I’m gonna
hopefully start that team building experience and bringing in IT and bringing in legal and 

discussing things that I not normally would have discussed with them. 

So, you know, it’s funny, you did say 60 % of organizations have been breached. 

Here’s the thing, I read a statistic the other day that a cyber attack will happen every
14 seconds in 2025. 

That number is crazy to me. 

Another number that was really staggering was cybercrime is projected to hit $10.5
trillion this year. 

So while it might have only been 60 % of organizations in the United States, you don’t get
to $10 trillion by not targeting everybody, right? 

It’s not they’re just targeting the 1 % of large corporations, right? 

They’re targeting everybody. 

To put that into focus here, $10 trillion, if you looked at that from GDP standpoints for
countries around the world, it would be the third largest country in the world. 

That is unbelievable. 

Cybercrime, third largest country in the world. 

That is unbelievable. 

Okay, I think it’s time to pivot now, right? 

So we’ve talked about a lot of the bad, a lot of the struggles. 

Let’s talk about how we shrink this 24 day timeline. 

Let’s talk about what customers can do to not only recover from these attacks, but respond
and maybe even emerge better. 

So we call it our four pillars. 

Very important. 

So first and foremost, you have to have a tertiary copy of data behind somebody else’s
infrastructure. 

So now you have a good clean copy that’s immutable and indelible. 

You have to have that place to restore it to, which is an isolated recovery environment. 

We call ours clean room. 

That allows you to uh do testing, which is first and foremost. 

We’re not going to shrink the 24 days, even applying these methods, if we don’t do the
testing with it. 

So you have to have a place to go and test, because you can’t interrupt production. 

You can also do forensic analysis there. 

So a lot of times, let’s say you’re not even in an attack, but you bring in a new
cybersecurity tool and you want to test it against your organization. 

You don’t want to just release some malware in your organization and see how it finds it. 

And you also don’t want to go in a sterile environment that doesn’t look or feel anything
like your production. 

So with Commvault Cleanroom, I’m able to recover my environment to a location that’s safe, 

and I can release a malware in there and I can see what that cybersecurity does. 

On the other side of that, if you are hit, like I had a situation once where it was not a
destructive attack, we got the clear, we could go ahead and recover stuff, and the United 

States government stepped in and they said, you cannot recover right now. 

We’ve never seen this bad actor and we’ve never seen this type of attack. 

So they wanted to do their own forensic analysis. 

Real quick, what was their SLA on that? 

Yeah, they blew out the… 

Yeah, of course, right? 

There is no SLA when the government’s doing it. 

No, not at all. 

It took us two weeks. 

Two weeks for the government to do their research. 

So wouldn’t it have been great if I would have had Commvault Cleanroom and could restore
those machines up to that environment and said, hey, take all the time you want. 

Go ahead and do your evaluations, figure out your stuff. 

We’re going to get back to business. 

Yeah, awesome. 

Very awesome. 

So then, so I have good data. 

I have a clean place that I can take the data to. 

The next and most important thing is cyber recovery plan, which we’ve talked about, which
is great. 

It’s going to detail everything, but there’s some things that have to go in that cyber
recovery plan that organically is very difficult to get. 

So that’s a discovery of all your application mapping. 

If I’m restoring just a couple files from a file server, I don’t really care. 

if I’m bringing up, let’s say, Epic for a health care environment, and it’s got 32
different servers that all interact with each other, and 

I’ve got to bring them all back. 

I’ve got bring them all back at a consistent state. 

And I hope you’ve done discovery to find that out, to do your application mapping. 

There are some tools that help you do that, but to be honest, it’s a very difficult
process. 

Yeah, how do you figure out 32 servers feed one machine, right? 

And most people are like, well, we know because we’re going to bring back our tier one
servers. 

Well, the problem is a tier one server is only workable if you have a bunch of tier three
and tier four servers that are actually feeding the data to it, 

that it needs. 

And so the only way to do that is with this fourth step, which is test with chaos. 

I’ve got to get into the clean room. 

I’ve got to make sure I’m doing recoveries, randomize it, see what happens, see what works
and doesn’t work after that randomization. 

All right, maybe only 32 machines were hit, but I’m going to have to bring back 142
because of all the dependencies there. 

Those are really the four things, the air gap, the clean room or isolated recovery
environment, 

a cyber recovery plan and then testing all of that because a cyber recovery plan is just a
piece of paper until you actually put it to the test. 

Absolutely, right. 

It’s just a document until you’ve put it through its paces. 

And some of the testing I would advocate for is for customers to do, you know, tabletop
exercises, you know, get in a boardroom, talk about if we get hit with ransomware, what 

are we going to do first, second and third? 

When are we even going to invoke our cyber recovery plan? 

But I would take it one step further, too, right? 

You actually want to do a hands on cyber recovery. 

Absolutely. 

Hey, I have this thing called AirGap Protect, I have this thing called Cleanroom, now
let’s put, the proof is in the pudding, right? 

Let’s make sure we can recover these workloads from this tertiary copy of data into this
Cleanroom, because that’s the true test, right? 

I can stand up my company. 

And that brings me to my next question is, how are companies even determining what they’re
gonna stand up for second and third? 

I’ve heard this term, minimum viability, thrown around a lot. 

I’m assuming that looks different for each organization. 

How does that fit in with these four pillars? 

It is different for every organization, right, depending upon what industry that you’re
in. 

And the only way you’re going to know it is through testing. 

It’s unknown without that. 

So many times people think that they understand what needs to go where and how. 

And without that testing, which we did for years for disaster recovery, so we have the
process. 

We’ve just not applied it. 

I mean, here’s the crazy thing. 

For years, 30 years, we’ve done quarterly testing for disaster recovery and disaster
recovery plans yet less than 1 % of all companies ever in existence have ever declared a 

disaster and as we said earlier 60 % of all companies in America had a breach last year
and no one’s testing for it. 

No one’s testing because it’s advanced, right? 

It takes a PhD to understand cyber recovery versus disaster recovery which everybody can
do nowadays. 

Sure. 

Yeah, and it’s almost like, hey, how do I even know what I’m testing in a cyber recovery
event, right? 

And I think that, you know, our customers here at Commvault are really lucky. 

We’ve now released this new service, this Guardian service, where, hey, we get that this
doesn’t happen overnight, right? 

If we wanted to be cyber resilient tomorrow, we would be, but it’s just not that easy,
right? 

It’s a journey. 

It takes time. 

In a lot of scenarios, you more often than not will need help. 

And I think that this new service is aimed at really helping customers 

take that next step in their cyber resilience journey. 

Well, and through our readi verse, because so many people don’t understand how to do a
proper tabletop exercise, we now have both an executive and a technical track of a 

tabletop exercise around cyber events. 

But as you said, that’s only the first step. 

Now you actually have to do it for real. 

In the military, we called it an OODA loop: Observe, Orient, Decide and Act. 

And what it does is it’s a repetition that you go through, 

of constantly testing something, evaluating it, changing it, testing it again. 

And that way you’re more prepared when something actually happens. 

Well, you’ve gone through so many cycles of testing and changes and ways of doing things
that it’s just another test almost to you. 

Sure. 

And this is how you really rise back up, right? 

So with these four pillars, the air gap protect, the clean room, your cyber recovery plan,
and then testing with chaos, right? 

And we have.. 

we align to those perfectly, like you said. 

We have the safe, secure copies of our data. 

We have the place where our data is going back. 

Heck, if you need a cyber recovery plan, go download it from the Readiverse, as I
mentioned, and make it your own. 

And then last but not least, if you need any help along the way, we’ve got you covered
with the Guardian Service. 

Michael, thanks for your time today. 

It was really insightful. 

Thank you, Chris. 

And that wraps up our conversation on the Phoenix Protocol and how Commvault’s four
pillars of cyber resilience really help customers rise from the ashes after a cyber 

attack.