Use Case: Dell EMC Avamar
Legacy Backup data migrations are typically perceived to be high cost, take a long time to complete and prone to error due to the manual effort involved. For these reasons and many others, customers have chosen to avoid migrating their data. They have chosen instead to wait for the data to expire (essentially aging off on the legacy backup system) and move forward with all new data managed through Commvault. However, there are instances where keeping the data on a legacy system is not a viable option; therefore data migration to the Commvault data platform becomes a necessity. These reasons are typically due to the following circumstances:
- Extended retention policies where customers do not want to maintain both systems for financial, risk, or labor overhead reason
- The customer wants to take advantage of Commvault’s modern data management platform for both new and legacy data
- Avoidance of costs associated with extending licensing as well as support and maintenance contracts associated with maintaining legacy infrastructure environments
To address the data migration barriers of risk, cost, time, along with the manual effort, Commvault has developed a solution to move data from legacy data protection software to the Commvault® data management platform. Commvault software interacts with the legacy platform to perform a data restore to a temporary storage location and then ingests the data by performing a backup, and then as the final crucial component, Commvault software automatically marks the data with additional metadata that preserves the original backup time, retention schedule and other key components of the legacy data. This whitepaper will describe in more detail the following:
- Industry trends requiring a cost-effective data migration solution
- An overview of Commvault’s Automation Language – Workflow
- A technical overview of the Commvault Data Migration platform with the use case of Dell EMC Avamar to Commvault
Unstructured data to rise from 9.3 zettabytes in 2015 to 44.1 zettabytes by 2020.
Unforeseen Vendor Lock-in with the Transition to Disk Backup and the Increased Need for a Cost-Effective Data Migration Solution
An industry shift in the backup and protection industry in the last few years has seen the transition away from tape as a storage medium and the increased usage of disk-based targets using deduplication technologies, especially for long-term storage requirements. Why does that matter for switching from legacy data protections to new software? Well, when tape was the primary medium for long-term data storage, you only needed to keep a minimal footprint of the legacy backup system to perform restores, which was low cost. Essentially, all that was needed was a server with legacy software installed and one or few tape drives to maintain access to all your data, while servicing the occasional restore.
With increased use of disk-based systems for long-term retention, you aren’t able to keep a “small instance” running to access all the data. The main difference with disk-based versus tape-based systems is that the entire disk-based system needs to be operational in order to access all the data at all times as compared with a small tape installation. In general, keeping an entire disk-based system requires substantial maintenance costs, especially in years five and beyond. Customers can feel “locked-in” to the disk-based system with the potential for escalating maintenance costs, which can leave customers feeling that their hands are tied, and potentially taking a “doing nothing” approach thereby staying with and accepting a sub-par solution.
Commvault’s data migration offering is focused on substantially reducing labor cost and risk through automating the steps required to migrate the legacy data.
Overview of Commvault® Workflow
The Commvault® Workflow engine enables rapid, reliable and repeatable automation of data protection operations. Workflow operations, including integration of third-party APIs, can be scheduled and executed from a command line or driven through a web process. This engine offers considerable flexibility for tailoring the end-user experience to the specific requirements of the business. Example workflow projects include:
- Billing system integration
- Application and Portal integration through C++/CLI/XML/REST/JAVA/HTML
- Object Link Deployment through REST
For the purposes of migration, Commvault has integrated with third-party legacy backup software (NetBackup, Evault, Avamar) to orchestrate restore of data, XML creation for metatags and ingestion into Commvault’s virtual repository.
Technical Deep Dive – Commvault Data Migration through a Dell EMC Avamar Use Case
The following sections will describe how Commvault can migrate data from legacy software to Commvault software using Dell EMC Avamar. The technical migration is separated into three distinct phases:
- Discovery Gathering data from the legacy environment as inputs to the migration by executing legacy software vendor specific commands
- Data ExportExecuting legacy software commands to restore data from the legacy environment to a target storage location (typically referred to as a ‘landing zone’)
- Data IngestionCreating custom XML for metadata and backing up the landing zone data into Commvault software
50% of IT teams are not ready for unstructured data growth.
Disaster Recovery to ANY Cloud
Cloud-based disaster recovery is quickly becoming one of the biggest use cases for the cloud. And at Commvault, we can help you uncover all of the cloud’s disaster recovery advantages.
Phase 1 - Discovery
Commvault needs to understand several key pieces of information in order to properly scope, estimate and plan the migration effort from the legacy backup environment. To gather the necessary information, Commvault software will either request scripts or request that a series of commands to be run on the source legacy backup software.
These scripts and commands will obtain the following information:
- Number of backup jobs
- Number of clients/servers/ VM
- Client names
- Client retention policy – 30 day, monthly, yearly, etc.
- Client original protection date
- Clients’ backup sizes
- Backup types – full, incremental, etc.
- Backup agents utilized for protection – file, system, database, etc.
- Backup sources – tape, disk, etc.
Depending on the legacy backup software vendor, Commvault may gather additional relevant data. In the Dell EMC Avamar example (figure 1), Commvault gathers additional data pertaining to the DOMAIN construct, which will be required to perform a restore.
Figure 1. Example Discovery Commands – Avamar
Management Console Command Line Interface (MCCLI) commands to be run on Avamar Server
$ mccli domain show --recursive=true > domain.txt
$ mccli client show --name=/XYZ-DOMAIN/XYZ-CLIENT > client.txt (Use an example client which would qualify for migration)
$ mccli group show --name=/XYZ-GROUP > group.txt (From the client show command output run for group client is a member of)
$ mccli retention show --name=/XYZ-RETENTION POLICY > policy.txt (From the group show command output run for the associated retention policy)
AVTAR commands to be run on example client in avs install location under folder bin
$ avtar --backups --server=XYZ-AVAMARSERVER --id=XYZ-USER@avamar/ --ap=XYZ-PASSWORD --path=/XYZ-DOMAIN/XYZ-CLIENT --retention-type=daily > daily.txt
$ avtar --backups --server=XYZ-AVAMARSERVER --id=XYZ-USER@avamar/ --ap=XYZ-PASSWORD --path=/XYZ-DOMAIN/XYZ-CLIENT --retention-type=weekly > weekly.txt
$ avtar --backups --server=XYZ-AVAMARSERVER --id=XYZ-USER@avamar/ --ap=XYZ-PASSWORD --path=/XYZ-DOMAIN/XYZ-CLIENT --retention-type=monthly > monthly.txt
$ avtar --backups --server=XYZ-AVAMARSERVER --id=XYZ-USER@avamar/ --ap=XYZ-PASSWORD --path=/XYZ-DOMAIN/XYZ-CLIENT --retention-type=yearly > yearly.txt
Note: Any word reference that includes prefix “XYZ-“should be replaced with the environment specific name
After the data is collected from the source legacy backup environment, the Commvault Professional Services team will be engaged to discuss migration options and requirements. For example:
- Does all of the legacy data need to be migrated or a just a defined subset (for example monthly full data or yearly full data only)?
- Estimated temporary storage requirements
- Defining in scope as well as out of scope data sets
- Schedule Pre-Migration Testing – the goal of this phase is to perform a migration of a
small sample of data before conducting the full-scale production migration effort to:
- Predict production migration timeline by gathering and reviewing performance metrics from the legacy backup software environment
- Harvest performance data from the legacy backup software environment to predict the migration duration
Phase 2 and 3 - Data Export and Ingest
Commvault purposely grouped together the discussion of the second and third phases to show the entire process through a series of execution steps. Below (figure 2) is a diagram depicting the high-level process with seven steps. For consistency, Commvault will describe the steps then show an Avamar example.
Figure 2. Migration Workflow
Migration Execution Steps
Commvault Workflow communicates with the legacy backup software to initiate and
orchestrate data movement to the landing zone. To initiate migration, the workflow can
be scheduled or manually initiated through the GUI – as noted in the screenshot below
(figure 3). The screenshot included below is an example of an Avamar use case workflow
as it’s requiring the DOMAIN variable, specific retention options, which are Avamar
- BackupServerName – Avamar node name or IP address
- ClientName – virtual machine / server that Commvault will orchestrate data restore
- Username – Avamar username that has the privileges to perform restore on specific client
- Password – corresponding password to username
- StagingPath – location of Landing Zone where data will be restored to
- Retention Options for restore – Not all data may need to be migrated, therefore various options are available depending on organizational requirement. For example, if only yearly backups are required, Commvault makes it easy to select just those jobs.
- Backup types – Different backup types may require different treatment and metadata tagging, therefore Commvault provides the ability to specify if a job is filesystem (UNIX or windows), Exchange, Microsoft SQL, or VMware. Avamar provides the ability to take a Microsoft SQL agent based backup and restore it as a flat file. By selecting the SQL option Commvault will orchestrate the Avamar commands to restore an SQL agent based backup to a flat file and then ingest it into Commvault.
Figure 3. Commvault Migration GUI – Avamar Example
Commvault Workflow executes legacy backup software commands to restore data from the
legacy backup software and target to the landing zone. The size of the landing zone is
generally substantially less than the total amount of data to be restored (an exception
is a situation where there are a small amount of jobs to be migrated, and they can all
be accommodated within a single landing zone platform). Commvault will determine the
landing zone size during the Pre-Migration Proving phase.
- Factors that affect the size of the landing zone include:
- Largest and average backup job size
- Total amount of concurrent streams from legacy backup software
- Average restore throughput from legacy backup software
Figure 4. Legacy Backup Software Restore Commands – Avamar Example
$ --labelnum – This info is pulled from the backup list command: “avtar --backups --server=10.201.104.11 --id=cvadmin --ap=******* --path=HAY1/file1.internal.net --retention=monthly”
$ FS - avtar -x --server=10.201.104.11 --id=cvadmin --ap=******** --path=HAY1/file1.internal.net --target=f:\ObjectStoreStage\CLientName --labelnum=1741 --tryagain=false
$ SQL - avtar -x --server=10.201.104.11 --id=cvadmin --ap=******* --path=SOC1/uk1sql_infra.serviceworks.local --restore-destination=single --target=f:\ObjectStoreStage\SQLClient --labelnum=9750
$ Exchange - avtar -x --server=10.201.104.11 --id=cvadmin --ap=****** --account=FST1/fstar-mail01.f-star.uk --labelnum=4979 --target=f:\ObjectStoreStage\EXCLient
$ VMWare - avtar -x --server=10.201.104.11 --id=cvadmin --ap=******* --path=/uk1soc-vc01/VirtualMachines/swc1/swc1-server01_UAIOS9FRvxOCooWJMxDeCA --labelnum=15 VMConfiguration\ VMFiles\ --target=f:\ObjectStoreStage\VMClient
- Each restore job from the legacy backup software is written to the landing zone with a specific folder structure to reflect client name, backup date, and backup time. Example folder would be represented as: CLIENTNAME_BACKUPDATE_BACKUPTIME
- Commvault Migration Workflow generates metadata for each restoration job and creates an XML file for metadata to be imported into Commvault software. Associating custom per job metadata provides Commvault the ability to maintain original backup date, data expiration, and other properties from the legacy backup software. Without custom metadata, imported backup jobs would show current date as protected times and would be extremely labor intensive to search for historical data. Also, Commvault utilizes the metadata to set data expiration at a per job level automatically, which would otherwise be a manual, labor intensive, and in some legacy backup software environments, impossible to do. It also introduces a level of risk, which can be avoided by effective automation.
- Data will be imported into Commvault with the Object Store client, which can be setup to run at regular intervals throughout the day. Once new data has been detected Commvault will read the XML file and begin the ingestion process. After the data has been imported, Commvault will delete the data in the Landing Zone. This removes the need to perform any post migration scripting or manual clean-up jobs after each import operation.
- Data is indexed asynchronously and custom metadata processed while under Commvault protection.
Once fully indexed data is searchable via Commvault’s Global Smart Index and accessible
through Commvault’s Web Console. Example search queries could be:
- When was the original date of backup?
- Which data expires in 2 years?
- Where is client’s data from 3 years ago?
- Are there any excel spreadsheets from client A that are 4 years old?
Figure 5. Commvault Migration XML – Avamar Example
<App_FileResourceListRequest> <parent directory=”1” path=”\ps-client-dev_2017-02-01_18-00-07” name=”ps-client-dev_2017-02-01_18-00-07”> <customProperties> <nameValues name=”ClientName” value=”ps-client-dev.cvlt.com”/> <nameValues name=”BackupSystem” value=”Avamar”/> <nameValues name=”BackupLevel” value=”monthly”/> <nameValues name=”BackupDate” value=”02/01/2017”/> <nameValues name=”EXPIRY_DATE” value=”1499434925”/> </customProperties> </parent> </App_FileResourceListRequest>
Figure 1. Example Discovery Commands – Avamar
For more information regarding Commvault Object Store and Migrating 3rd Party Data, please visit our website: Migration of Third-Party Data to ObjectStore - Overview
Historically, organizations make software and hardware platform choices based upon a number of factors, but these can be typically summarized as technical capability, availability, integration capability, enterprise strategy and price. These decisions are made with the best of intentions and based upon data and information that is available at the time. However, times change and with it so does technology and capability; and when we reflect upon the ever changing needs of the business along with changing market conditions and industry trends, we find ourselves needing to adapt and evolve. Sometimes that evolution can be hampered and made complex when you have to consider moving and migrating data and workload from one platform to another.
Migrations from legacy backup platforms to Commvault can now be made simpler, faster, and more cost effective with less risk by leveraging Commvault’s automated migration solution.
Organizations will no longer have to retain, maintain and manage a reduced footprint of their legacy software platforms purely for long term retention restores, nor will they have to wait around for their data to expire; you can plan to move to Commvault today and begin benefiting immediately from Commvault’s industry leading enterprise data management platform. Organizations no longer have to feel “locked-in” or committed to a legacy backup vendor, as Commvault can help and support making the switch planned, predictable and with low deployment risk.
So why don’t you take the next step in moving to a modern, flexible and singular enterprise solution? Unlock your data from your legacy provider, and move to Commvault’s modern data management platform!