Modernizing Commvault’s Architecture: From Monolith to Scalable Microservices

Across businesses in every sector, data volumes are expanding exponentially. Organizations are collecting, processing, and acting on larger and more diverse data sets than ever before. With this relentless data growth, there emerges a new challenge: How can existing data infrastructure keep up, not just in size, but in flexibility and intelligence?

Commvault recognizes that growth demands change.

Initially, Commvault was built for the world of on-premises data center protection. Our products were designed to scale vertically but not horizontally, which we realized posed an issue for our customers.

Our foundational services operated on single machines, which we learned created a bottleneck that limited both flexibility and resilience. More memory and faster processors were able to extend capacity; however, this only proved to be a temporary solution.

As we built our cloud products, we had to evolve and embarked on a massive architectural shift that embraced horizontal scaling, microservices, and a platform foundation built for today’s data workloads.

Recognizing Our Limits: The Inflection Point

 System limitations and monolithic architecture created a negative user experience for some of our customers. So, we started making changes.

One core functionality that we started scaling up first was MediaAgents, which are software components that act as data transmission managers, handling data movement and storage management across multiple machines. With those being some of the most data-intensive components, they struggled to keep up as data loads increased.

The solution was to scale-out horizontally, allowing users to bring in more machines to distribute the load. This change marked the beginning of horizontal scaling across the platform. By letting users add machines rather than upgrade the same one, we introduced flexibility. MediaAgent proved that a horizontally scalable approach worked and the rest of the platform should follow suit.

Laying the Groundwork for Scale: Introducing the Entity Lookup Service

As we worked to improve our data movement layer, we also decided to focus on another critical dimension of scale: performance of the Command Center.

Commvault’s platform is inherently write-heavy, continuously ingesting data for discovery, job tracking, auditing, and more. While this model worked well with relational databases that offered strong consistency and transactional guarantees, read performance could not keep up in large environments.

We introduced the Entity Lookup Service to improve the user experience in the Command Center, where some API calls were taking 30 seconds or more to respond.

This lightweight, read-optimized service maintains a denormalized copy of key relational data in a document-based store such as MongoDB, which allows us to achieve our core performance goal: keeping all read-heavy API responses under 3 seconds, regardless of scale.

This architectural shift not only dramatically improved user experience but also laid the groundwork for further platform optimizations, some of which we detail below.

Rebuilding Our Capabilities: The Role of the CVDotnetContainer

With the successful evolution of MediaAgents, we recognized that our other core components also could be expanded. Our base platform services, such as the Job Manager and CommServe server, still utilized a single machine. This created architectural rigidity and created limitations on our ability to effectively manage growing workloads, regional requirements, and/or customer preferences.

To meet these demands, we decided to shift toward a microservices-based architecture to break large services into smaller, purpose-built components that could be developed, deployed, and scaled independently. This approach provided greater flexibility and more efficient use of infrastructure.

However, it also introduced new challenges. We needed a way to deploy multiple services without consuming excessive resources, concurrently support the coexistence of diverse services on a single machine, and give administrators the flexibility to choose where each service would run. This called for a system capable of hosting modular components while managing shared platform responsibilities.

The solve for this challenge became the CVDotnetContainer.

Making Microservices Practical at Scale

Built from scratch, the CVDotnetContainer acts as an effective hosting layer that supports multiple microservices on a single instance. It provides the flexibility to deploy services independently, based on the needs of each CommCell. Once the base layer was prepared, key services such as S3 Integration, App Manager Lite, and Audit Service quickly adopted the container to run on any machine, with minimal effort and full flexibility.

Beyond its hosting capabilities, the container takes care of common operational requirements. It handles authentication, request and response lifecycles, error management, logging, registry access, and file system interactions. Developers no longer need to rebuild these foundational layers for every new service. Instead, they can focus entirely on the application or business logic.

The CVDotnetContainer brought modularity to Commvault’s platform, streamlined deployments, and allowed services to scale independently. As a result, our overall architecture became more maintainable and better aligned with the demands of ever-evolving modern workloads.

Simplifying CommCell Management: The Global Command Center

As enterprises started reaching global audiences and more data started flooding in, a single CommCell environment proved to be inadequate, given it only has a single instance of a CommServe server. These organizations began setting up multiple CommCells across different regions to meet regulatory, operational, and performance requirements. These instances operated independently, which meant managing them became a fragmented experience.

As we embraced horizontal scaling to enhance user experience, the management of these distributed environments presented Commvault with a new challenge. In response, one of the most significant advancements was made to the Commvault architecture, enablement of a simplified management of multiple environments in a single control plane with the introduction of the Global Command Center, which allows customer to manage all of their CommCells.

The Global Command Center is a control plane that serves as the “single pane of glass,” allowing customers to manage multiple CommCells through a single administrative interface. It created a consolidated view of operations, even when infrastructure was spread across continents. Customers could deploy instances in the United States, Europe, Asia, or any other location, and still monitor and control them from one place.

This unified view eliminated the need to log into each console separately. Instead of requiring multiple administrators to maintain CommCell environments, this single global platform enables customers to monitor jobs, track updates, and oversee the entire enterprise’s health. It helps simplify oversight, reduce operational burden, and allow teams to manage global ecosystems with the clarity of a centralized system.

What Lies Ahead: Designed to Help Create a Future-Ready Enterprise

Commvault’s evolution into a microservices-based, horizontally scalable architecture has fundamentally changed the way we serve our customers for the better. By embracing modularity, providing flexibility, and introducing centralized management through the Global Command Center, Commvault has designed a platform ready for the requirements of today and to help meet the demands of tomorrow. No matter how complex modern workloads become or how large data volumes expand to, Commvault is designed to provide exceptional user experiences.