Pittsburg State University (PSU)
Maximizing business agility with Hedvig
Many of PSU’s faculty and staff live just across the state line, in or near Joplin, Mo. On May 22, 2011, a mile-wide F5 tornado tore through Joplin, killing 158 and causing $2.8 billion in damage. Several organizations in Joplin with hardened data centers sustained damage from the 200-mph winds packed inside that F5 mega-vortex. Some of them were offline for significant periods of time.
“This was a wake-up call to us at Pitt State,” said Tim Pearson, Assistant Director in charge of IT Infrastructure and Security at PSU. Until this time, PSU employed what most would consider “standard” methods of data protection and security. The Joplin tornado made it abundantly clear to the IT professionals at PSU that “standard” wasn’t good enough when you live in Tornado Alley!
PSU began a process to distribute its compute and storage infrastructure to improve survivability should an event similar to the Joplin tornado strike Pittsburg.
“In addition to moving half of our production VMware compute stack to our disaster recovery (DR) site on the other side of campus, we purchased some used EqualLogic gear to replicate our storage to the DR site multiple times each day.
“We also leveraged the State of Kansas Research and Education Network and relocated some old storage to the data center at Wichita State University across the state. We replicated our backups to Wichita every night.” These actions significantly enhanced the survivability of PSU’s virtual environment using the then-available technology and budget.
It wasn’t until there was a significant increase in the failure rate of the spinning disks in its old EqualLogic SANs that Pearson began to raise red flags to university leadership about the aging condition of some of the university’s most critical IT infrastructure. University President Steve Scott understood the risk the university was facing if it didn’t act and identified. Federal grant money to fund the upgrade.
- In the heart of Tornado Alley in the Midwest
- Like most universities, had put off upgrading virtual infrastructure due to economic downturn, the tepid rate of the ensuing recovery and continuing ripple effects it was having on state revenue all over the U.S.
We deployed a three-site VMware clustered solution that significantly increased our resiliency and survivability, while delivering a payback on our investment in less than 13 months.–Tim Pearson,Assistant Director of IT Infrastructure & Security | PSU
PSU task force’s infrastructure objectives:
- Enhance data center fault tolerance and geographic diversity
- Enable incremental growth/linear cost
- Eliminate single points of failure
- Provide ‘always on’ (active/active) load balancing and failover
- Enable “point-in-time” data protection (snapshots/backup)
Pearson created a task force consisting of System Administrators, Datacenter Admins, and Senior Network Engineers and began the process of developing an RFP for new compute and storage. The task force conducted independent research on current technology, identified the players in the market, and determined the kinds of features the various vendors had to offer. It also identified some design goals for any new infrastructure. These included enough compute horsepower to run PSU’s critical VMware workload in both on-campus datacenters, along with real-time replication and automatic failover of the storage layer between at least the two on-campus datacenter locations — and preferably to Wichita State as well. In addition, external iSCSI and/or NFS connectivity was required to allow PSU’s other stand-alone Unix and Linux hosts to use the new storage.
“It was at about that stage in the process that my research helped me find Hedvig,” said Pearson.
Pearson downloaded several technical documents from Hedvig’s website and began to familiarize himself with the design philosophy and feature set Hedvig offered.
“I knew after 10 minutes of reading that Hedvig was a serious contender for our storage layer,” said Pearson. “[It] offered such an elegant, robust, scalable, and flexible distributed storage model. It really put them in a class different from most of the other vendors out there.”
As part of the RFP evaluation process, Pearson’s task force paid special attention to vetting Hedvig’s track record with its customers.
He explained: “Hedvig arranged for us to speak with several of their longer-term customers whose use cases were similar to what we wanted to deploy. The most memorable outcome of those conversations was the universally glowing reports we got on Hedvig’s support, along with statements about their willingness to let customer needs influence the enhancement process as new versions of Hedvig were developed.”
Hedvig’s proposal eventually won the day at PSU. “The members of our evaluation task force unanimously recommended Hedvig,” said Pearson.
Hedvig’s solution not only satisfied all the requirements of PSU, but also gave it a glimpse of what it can achieve in the future. With Hedvig’s object storage capability, backup integrations, container support, and hybrid-cloud/multi-cloud deployments, PSU insured that their solution is future-proof, as well.
PSU implemented a highly available, private cloud solution for their VMware environment across three physical sites that had a payback period of approximately one year compared to renting the equivalent capacity in the public cloud. Specifically, PSU spent USD $225,000 to acquire the distributed storage software and another $84,000 for a new ESX host in the first year, for a total investment of $309,000. The equivalent cost to rent public cloud resources would have been approximately $285,000 ($23,700/mo.) for the first year, resulting in a payback of less than 13 months.
Since the technology was new to PSU, it took a deliberate approach to rolling it out. It started by moving half of its ESX environment to a second location. Initially, it conducted manual failovers to this second location, and eventually to the third location until it was comfortable with the process. Now, it’s all automated.
“It took a while to get over the psychological impact of geographic separation, but now we don’t think about it,” said Pearson.
Similarly, PSU started with the backup target use case and eventually expanded to its VMware workloads. Currently, it is deploying multiple workloads across three geographically dispersed sites in an active/active/active deployment, and it uses Commvault Complete™ Backup & Recovery software for ‘traditional backups’ as an additional form of protection.
The combined solution provides predictable pay-as-you-go scaling, real-time fault tolerance, faster recovery and simplifies management by providing a single solution across the entire environment that is managed by the same team that manages the server, network and storage infrastructure.