By Matthew Magbee
If you, like me, are responsible for disaster recovery and business continuity at a global healthcare organization, there are few things you dread more than a successful ransomware attack. The recent WannaCry offensive, which affected healthcare in the UK, demonstrated that. If your disaster recovery plan or data management platform are not up to the task and fail after such an attack, you might find yourself paying a ransom or devoting thousands of man-hours to manually rebuilding the data that has been encrypted. Even worse, if your organization’s clinical, EHR and other applications or data are inaccessible for an extended period of time (or any period of time, for that matter), doctors and nurses might not be able to access critical patient information – and not just your business, but also patients’ health could be threatened.
Therefore, it is no surprise that my heart skipped a beat the moment I learned a ransomware attack had succeeded in infecting one of our clinical imaging file servers hosting patient lab results. A Globe2 ransomware infection had infiltrated our environment through a connection to a third-party vendor’s clinical system, which hadn’t been kept up to date. But we had a plan for this, and had tested that plan (again and again). Moreover, we had a corporate policy that we would never negotiate with cybercriminals. Now the moment of truth had arrived to activate our carefully developed disaster recovery plan and quickly regain control of our data.
We traced the source of the attack within minutes, found the rogue system, killed the account and wiped the infected drives to contain the spread of the worm. Seeing all that data deleted from the drives before we restored the data was scary, but it was the only way we could ensure that the worm would not infect any other systems. We used Commvault software to restore both file data from the previous night’s backups and to recover the database from a snapshot we had taken earlier that morning. Our applications were up and running within 30 minutes, and then we began restoring the data. All but 20 minutes’ worth of data was restored within less than five hours – and all this was accomplished through a single platform solution. We still needed to rerun a minimal number of clinical samples that had been tested before the attack and rush the results to waiting doctors, but there was no impact on the health of a single patient. I can’t say the experience was not a harrowing one, but the plan we designed for exactly this type of situation went off without a hitch.
Given this incident, and my other disaster recovery experience, here is my advice on how other healthcare data management professionals can prevent a successful ransomware attack – or any other data disaster – from becoming a major problem. I can’t provide you with all of the answers, but here are four key recommendations:
- Be prepared with a plan: You might think it can’t happen to you. Don’t fool yourself. Ransomware attacks are not only growing in number, but are becoming more sophisticated as well. Innovation is not limited to the ‘good guys.’ And, of course, your next disaster might not be ransomware related; it could be a hardware failure, a natural disaster, or something else. So make sure you consider all the possible disaster scenarios, and have plan and systems in place that can restore your data for each type in the time needed by your users. Then be sure to update your plan on a regular basis. Your system and the threat environment will change. Your plan needs to adapt to these changes as well.
- Test, test, and test again: Your plan looks great on paper. But will it work when it’s crunch time? Test your plan on a monthly basis, and make sure it works for everything – files, emails and VMs. Test restores for different periods, with all your backup media – disk, tape, etc. Not only will you ensure your plan is battle tested, but when you activate the plan you will know exactly what to do and how to do it.
- Read your reports: You might think you have better things to do than read reports. You don’t. While this might have been the first major ransomware attack I have had to deal with at this organization, it certainly has not been the first time I have had to restore data – not by a long shot. The more you know about your environment, the easier it is to find holes and repair them before they become serious issues. So read your reports every day. Take it from me.
- Use Commvault solutions: Speaking of reports, Commvault’s reports have saved my rear more than once. Yet that is just one of the reasons I trust Commvault software to protect and manage our data. Commvault provides you with a single, comprehensive, powerful platform for managing all your data, whatever type of data it is, and wherever it may reside. This does not just make backup easier, but also simplifies other tasks like archiving and data governance, helping you reduce the time you need to spend on things like HIPAA compliance. And don’t even get me started on Vault Tracker. I don’t want to even think about trying to manage off-site copies without it.
I can’t promise you that if you follow the four recommendations above that surviving a ransomware attack will not be a nerve-racking experience. It certainly was for me. I can tell you these recommendations are a good first step toward making sure that you are doing everything you can to protect your patient data, and prevent any delay in the timely delivery of care to folks who are ill.
Matthew Magbee is disaster recovery and business continuity engineer at a global healthcare company with a reputation for excellence in laboratory medicine and pathology, radiology and diagnostic imaging and primary care medical services.