University of Rochester Medical Center
Enhances patient care with improved management of electronic medical records and research data
- Outgrew legacy platform in 2.5 years, despite five-year growth plan
- 75% of backups failed
- Unreliable backups impacted strict four-hour SLAs for data restores
- Reliable, scalable platform was required to keep pace with aggressive data growth of up to 50% annually
- Commvault software with capacity licensing model, deduplication and Virtual Server Agent to accommodate heavily virtualized environment
- Unified data management platform along with accelerated backup and recovery of patient records and research data supports URMC’s meaningful use initiatives while ensuring the highest levels of patient safety and medical care
- Scalable platform accommodates rapid growth while enabling increased protection of crucial patient records, medical files and research data
- Global deduplication with ratios of up to 80% reduces storage costs and eases administration
- Commvault’s Virtual Server Agent enables acceleration and elevation of virtual server protection; plans to add SnapProtect will speed restores of vital medical data
The University of Rochester Medical Center (URMC) is one of the nation’s top academic medical centers and the centerpiece of the university’s health research, teaching, patient care and community outreach efforts. As an integrated academic health center, the university’s healthcare delivery network is anchored by Strong Memorial Hospital, a 739-bed teaching hospital with programs consistently ranked among “America’s Best Hospitals,” according to U.S. News & World Report.
With more than $145 million in federal research funding, URMC ranks in the top one-quarter of U.S. medical centers in research funding, while its School of Nursing ranks 12th in funding nationwide. Additionally, URMC has achieved top 15 rankings in National Institute of Health (NIH) funding in biochemistry, microbiology and immunology, neurology, obstetrics and gynecology, oral biology and musculoskeletal research.
The faculty of URMC’s School of Medicine and Dentistry, which consists of roughly 1,400 full-time and 650 voluntary clinical members, joins approximately 900 specialty primary and attending physicians, 400 medical students, 500 graduate students as well as 600 residents and fellows in helping patients benefit from the medical center’s robust teaching and biomedical research programs.
An overarching mission to use education, science and technology to improve health and transform the patient experience permeates all aspects of URMC, including the IT group that oversees the medical center. This 280-person staff embraces innovative, leading-edge IT technology to support sophisticated solutions for electronic medical records (EMR), Picture Archiving & Communication Systems (PACS), imaging and hospital information systems, patient scheduling and billing, and much more for two hospitals, 121 locations and up to 160 outpatient clinics.
According to Rick Haverty, director of infrastructure for URMC, technology plays a vital role in supporting the continuous demands of the medical center’s many caregivers. “It is critical for us to support the center’s core mission, especially in managing IT systems that drive healthcare delivery, all of which need to run non-stop, 24/7, 365 days of the year,” he explains.
- Microsoft Windows
- VMware vSphere ES
- Microsoft SQL Server
- Forty-two critical medical applications, including Epic, GE Flowcast, McKesson Star, Allscripts and more
- Hundreds of additional applications to support medical and dental information systems
- IBM System Storage DS8000 Series
- IBM XIV Storage System
- IBM Storwize V7000 Unified Disk System
- IBM 5020 mid-range arrays
Commvault software enables us to fully protect 42 applications that are essential to patient care along with hundreds of other applications that are also important in running our operations.– Rick Haverty | Director of Infrastructure, URMC
Unlike previous dealings with our legacy vendor’s support organization, Commvault’s BCS team was very responsive, knowledgeable and eager to work with us to streamline our deployment and help us realize measurable benefits immediately.– Rick Haverty | Director of Infrastructure, URMC
Data management environment
Approximately a dozen members of the IT group ensure that URMC has state-of-the-art file servers, storage and data management solutions to support its ever-increasing cadre of essential medical applications. Currently, the team supports a mix of 300 physical file servers running Unix and Windows 2008 and 2003 along with 600 virtualized file servers running VMware. “We look at each new server request to first see if it can be virtualized to reduce capital costs, space and power,” adds Haverty. “We’ve also found that our server virtualization strategy eases disaster recovery and maintenance. To that end, we really appreciate how Commvault software helps us scale out our virtualized environment while easing support and protection of our legacy physical servers.”
On the storage front, URMC takes advantage of Commvault technology to manage 180 TBs of patient records and research data that resides on multiple tiers of storage to meet various retention requirements. On the first tier, data is stored on an IBM System Storage DS8000 Series array and then replicated to another DS8000 in the second data center. Tier two storage resides on IBM’s XIV Storage System while tier-three storage is relegated to IBM’s Storwize V7000 Unified Disk System. Backup data is stored on a pair of IBM DS5020 mid-range systems before it’s moved to tape for long-term retention.
Overall, the medical center’s storage requirements are increasing by up to 50 percent each year as healthcare continues to go digital in keeping with meaningful use requirements and modernization initiatives as specified in the American Recovery and Reinvestment Act of 2009 (ARRA). Additional growth is anticipated as URMC centralizes certain technologies to ease decision- making and ongoing data management. A prime example is the impending move to a Vendor Neutral Archive (VNA) to consolidate storage for disparate PACS platforms located throughout the medical center into one back-end system to bolster centralized data management. “The challenge in healthcare is the abundance of highly customized workflows,” notes Haverty. “However, we constantly look for ways to manage data and simplify access to the benefit of everyone.”
Another consideration for URMC is accommodating different retention requirements, which can also impact storage dramatically. For instance, retention for a radiology file is seven years since it was last viewed, which can extend the initial retention period exponentially since follow-on viewing is commonplace. Other retention demands are dictated by audits from the National Institute of Health (NIH) and Joint Commission on Accreditation of Healthcare Organizers (JCAHO) as well as drug companies and funders reviewing data related to research projects and clinical trials.
Healing broken backups
Initially, URMC relied on Symantec NetBackup as part of a five-year plan for managing and protecting a variety of mission-critical applications encompassing patient records and research data. In less than three years, however, the team began enduring performance problems and significant failures. “Backups were failing nearly 75 percent of the time,” recalls Mike Nardi, server administrator for URMC. “We talked to our reseller and the vendor about ways to minimize backup failures but didn’t get good responses.
The URMC IT group conferred with the IT group responsible for the university as they also were experiencing similar problems with their legacy NetBackup platform. In particular, the vendor’s advice to buy more hardware wasn’t received well by the collective team, so they decided to look at new vendors who could address the persistent performance and reliability problems. Additionally, the team wanted stronger control over the backup and recovery environment so they could accommodate the high availability requirements of critical systems. “We have very strict service level agreements (SLAs) for more than 40 of our most critical applications,” says Haverty. “Target recovery times for eRecord, our Epic-based electronic medical records system, along with GE Flowcast, McKesson Star and Allscripts, is four or fewer hours as these are deemed vital to healthcare delivery.”
In seeking innovative solutions to better manage and protect crucial data, URMC turned to trusted industry analyst firm Gartner. URMC asked Gartner about other legacy platforms, such as IBM’s Tivoli Storage Manager and EMC Networker, before learning about Commvault software, which they were told was an “up and comer” that was taking market share from the legacy players. URMC then issued an official RFP, giving each vendor an opportunity to showcase product functionality and features ideally suited for a healthcare environment.
Commvault’s cure-all for safeguarding data
URMC’s RFP asked for lots of options for backing up and recovering different types of data as well as recommendations for reducing storage consumption in keeping pace with aggressive growth in patient records, medical files and research data. The team also asked each vendor for insight into future roadmap items that might be of interest, given the medical center’s reliance on digital data to support both research and healthcare delivery.
All the RFP respondents recommended deduplication as the solution for eliminating redundant backup and archive data while reducing backup windows. In sizing up EMC’s tactic, the team didn’t like the multi-prong approach, which included Avamar and Data Domain for deduplication on top of Networker for data protection. “We didn’t want a multi-product solution to perform one overall function,” notes Nardi. “It seemed like asking for trouble.”
In contrast, URMC liked Commvault’s source and target-side deduplication, which was embedded in Commvault software to reduce storage costs, use fewer network resources, shorten backup windows by up to 50 percent and eliminate the lock-in associated with appliance-based approaches. “Equally important was noting that Commvault offered edge protection functionality,” Haverty adds. “Since our desktops and laptops hold a lot of valuable research data, the opportunity to extend the same high level of data protection to these dispersed systems was appealing.”
As part of its due diligence, URMC called references for each RFP respondent and was pleased to learn that customers across the board reinforced the favorable positioning by Gartner. Another plus in Commvault’s favor was the fact that the IT department that oversees the university’s academic data grew leery of simply refreshing its own legacy NetBackup platform and also participated in the RFP process. They too felt that Commvault software was the best fit overall for its backup and recovery needs, so the teams joined forces to leverage Commvault software for safeguarding both academic and medical data.
Healthy, scalable data management prognosis
Assisted by Commvault’s Business Critical Support (BCS) services team, URMC deployed Commvault software to reap the full benefits of unified data protection and integrated deduplication. Source side deduplication has enabled URMC to drastically reduce redundant data across different storage systems and policies.
“We’re achieving dedupe ratios of up to 80% on Unix, Windows and SQL data,” says Justin Woodhouse, server administrator for URMC.
Thanks to Commvault’s dedupe capability, URMC has been able to standardize storage policies, especially on Windows, which eases ongoing administration.
The team also takes advantage of Commvault’s unified platform and centralized management console to accelerate both backup and recovery of 180 TBs of data spanning every department and medical specialty at URMC’s hospitals, clinics, physician’s offices, labs, cancer centers, etc.
“Commvault gives us a quicker, smarter way to backup data, so when we come in the next day, everything is all right,” says Nardi. “We’re now exceeding our recovery SLAs as a result.”
Previously, it took URMC up to three days to complete some file-system restores. With Commvault software, restores take place in a matter of hours, which gives the team a lot of confidence in ensuring the highest levels of data protection.
“Now, we get a ‘wow, this is back’ response to restores, especially in dealing with Exchange files,” Nardi adds.
Commvault’s ability to facilitate upgrades with “push updates” is a tremendous time-saver. “The productivity increase, coupled with better resource utilization, has led to impressive results,” says Woodhouse. “It used to take a couple of days to perform an upgrade and now we can complete the process in half the time.”
Since opting for Commvault’s capacity licensing arrangement, URMC can also easily use new Commvault software features as needed.
“Commvault software has lots of bells and whistles,” Haverty says. “We plan to use them when appropriate to help our continuous improvement efforts.”
A solution that allows your company to innovate and grow while keeping the business running uninterrupted, proactively enabling rapid change and applying effective governance in the face of regulatory shifts.