Pros and Cons – Rolling Your Own Backup Scripting

Posted 08/18/2016 by Commvault

We’re doing a lot of expansion and hiring in our APAC region right now, and one of the really fun parts of that process is helping on board some really talented new pre-sales and support engineers to our team. We got into a pretty lively discussion recently about writing shell scripts and wrappers. While it might make sense to try to use a pre-packaged solution to complete a complex backup that involves multiple layers and applications, versus 'rolling your own' solution with any number of languages and APIs.

Some of the engineers who had recently come to us from other backup companies were keen on the latter. They wanted to show me how easy it was to write their own programs that handled scheduling, inventory and tape control of the backup engines, hypervisors and other applications. I have to hand it to some of them; they are bright people who wrote some good scripts that worked really well. We hire some smart folks!

I complimented them on their cleverness and I asked them how they would feel about staying in one job all of their lives, with no chance of promotion? And how they would feel about using the same hardware and software for the rest of their careers? They looked confused. Also a little annoyed with me.

Maybe I was being a bit too cheeky with them. I had just read an interesting piece by Curtis Preston at Storage Switzerland called “Bats Aren’t Blind & IT Shouldn’t be Either.” Curtis was writing about VM snapshot automation and cloud migration tools, so maybe I was already a bit too amped up that day.

But in the 25 years or so that I’ve worked in the data center, I’ve had the good fortune to have met and learned from some really smart people; I’ve also seen some clever automation programs written and implemented. The problem with all of them was this: once these 'one-off' solutions got put into place, the person who invented them had to stick around and maintain them forever. I’ve seen it in dozens of places - there is some absolutely burned out sysadmin who spends his days maintaining some brilliant labyrinth of custom scripts that only he understands and that only he can maintain. He spends all of his time updating those scripts, trying to keep up with version updates and security fixes for the scripting language and all of the applications in his ecosystem. Think about that list for a moment: Hypervisors, SAN/NAS firmware, databases, CRM applications, ITIL service desks and CMDBs, backup and archive software…. You get the idea.

There’s precious little time for that sysadmin to learn new technologies or to do something else in his career, as the care and feeding of his custom automation software becomes more and more of a full time job. The thing that was originally supposed to be a time saver can become more cumbersome than the original problem that it was designed to solve.

We believe that the days of having to write lots of custom middleware and shell script wrappers to perform backup and archive tasks are over. That a modern data management strategy should include a robust backup engine that has the ability to natively communicate with lots of third-party hardware and software with simple radio buttons and drop down menus.

For more complex integrations, open APIs that use standard protocols - such as REST for cloud integration - should facilitate command communication. And built-in workflow automation tools should handle the logic and decision making.

The idea is that a well-trained staff should be able to learn a commercial application and be able to make changes to it quickly and easily, leaving the worry about compatibility between versions and hot fixes to the fleet of full time programmers and support staff who work for that commercial software vendor.