Why We Think Ovum Got Their Decision Matrix So Wrong

Posted 11/08/2016 by Commvault

  • Draws conclusions on products not yet shipped to customers

  • Simply incomplete – the biggest industry players weren’t included

  • No apparent depth of customer references

  • Ovum acknowledged the deficiencies above ahead of publication and proceeded anyway

Last week, industry analyst firm Ovum announced the release of its Decision Matrix: Data Availability and Protection Solution for the Cloud Era, 2016-2017. We'd have to say, we are a bit taken aback at the lack of completeness and discipline applied to this report, and we need to provide some needed balance.

We've always applied a simple test to industry analyst reports and to company press coverage – Is it both fair and accurate? To us, accuracy is a simple black and white thing – it either is, or it isn’t.  If it isn’t accurate, our responsibility in communications is to set the record straight. Fairness is much more subjective, and requires a finer examination of the issues.  

When we apply these simple tests of fairness and accuracy, in our view, we find Ovum’s work significantly lacking and simply incomplete on some of the merits. As such, we believe customers, prospects, partners and others should seek more research and counsel to ensure they are making an informed decision that is based on a more holistic view than gathered from this report.

Consumers of these reports typically expect a few things: First, they want to understand if the products being reviewed are real and delivered to the market. Second, readers presume the analysts have talked to verifiable reference customers who are actually using the products. And, third, that the analyst firm is presenting a reasonably complete picture of the market choices available. 

Let’s agree: customers, prospects and others rely on industry analyst reports for a third-party perspective to help them make technology decisions. These third-party analysts are supposed to be knowledgeable and independent, and help the reader get a complete understanding of the product space, the applicable vendors and technology trends.  

In our view, this latest Ovum report has a few glaring weaknesses:

  • The report takes into account products that are not currently in the market.  As an example, Ovum identifies Veeam as the clear leader, but evaluates their offering based on products and features not yet shipped to customers and not yet available in the market. This is a critical oversight, made worse because it violates Ovum’s stated methodology: “The criterion for a vendor to answer “yes” to a feature is that it must be available out-of-the-box in any product within its range of products that are applicable to its data availability and protection solution.” We challenged Ovum on this enormous miss – they responded that “it’s up to the vendor to respond honestly.” We found it very curious that after we raised concerns, the following language showed up in the Commvault section: “Commvault is also the most consistent vendor in the ODM, maintaining its position (leader) from the 2014/15 ODM. In terms of capability, Commvault’s submission is based on currently available solutions, not to-be-released solutions like some of its competitors.” One wonders if that wasn’t an olive branch of sorts.

  • The report does not include evaluations of this market’s two largest vendors – EMC and Veritas – and therefore is simply incomplete. From what we know, EMC and Veritas did not complete applications for Ovum’s evaluations. We can conclude they were too busy with being bought by Dell (EMC) or being spun out to private equity (Veritas) and couldn’t be bothered, or perhaps they simply missed the deadline, or maybe don’t care about Ovum’s work. They might not care, but we at Commvault want to see it done right, and Ovum should want that too. For Ovum to publish anyway, and to simply note that they declined to participate reduces the practical value of the report significantly.  A more valuable and comprehensive approach would have been to use their analyst “expertise” and publicly available information (including product documentation made available by EMC and Veritas) to predict or estimate the impact of EMC and Veritas on the Matrix charts (especially because these two vendors are significantly larger than all others evaluated).”

  • On reference customers, the report to me feels inconclusive. Ovum requires only one reference customer, and that customer must have more than 200TB under management.   Most of the industry analysts we work with at Commvault require far more than one reference customer, and I’m confused how vendors who were evaluated by Ovum with solutions not yet shipped to market could have met this criteria.

There are many industry analysts firms – the names are familiar, including Forrester, Gartner, IDC, the 451 Group, and, yes, Ovum, among many others. They all use different methodologies and even present their results differently – Ovum uses a “matrix,” Forrester has “Waves,” and Gartner uses the “Magic Quadrant.” Technology buyers have a lot of sources of information, and, based on the factors listed above, I’d certainly recommend looking beyond this particular Ovum piece of work. 

Three closing thoughts:

  • Ovum chooses to rate companies highly because of “market impact.” However, market impact doesn’t translate to customer value, and we'dd rather see Ovum put more emphasis on innovation pipeline, vision, ability to execute and breadth of offering already available to customers.

  • Let us be clear: this is not about Veeam; it’s about this particular Ovum report and one employee's point of view about this report. Sure, Veeam has touted their leadership because Ovum’s report says so – We think that claim in this case is based on an incomplete report, and doesn’t hold water. Miranda Lenning, who leads the analyst relations practice, has done an analysis of Veeam in a related blog based on the Ovum report – you can find that in her blog. That being said, we think Veeam deserves due credit for growing fast, and offering some good products, especially for mid- and small-market customers, and aggressive marketing. Good for them, but we think Commvault has a stronger enterprise offering, on many levels.

  • Most industry analysts fact check their reports before publishing, and provide formal processes for feedback and reaction. When we looked at the draft Ovum matrix, we spoke with the report author and raised valid concerns about the report (as noted above). Ovum acknowledged the issues, indicated the report was “already baked” and on deadline, and then published anyway. That, in my view, was disappointing.

This particular report may have challenges, but Ovum has a solid reputation and we know they do a lot of good work in general. We wish they’d have been more complete here, but nobody should conclude I’m “anti-Ovum” – to the contrary, we think they are a solid industry analyst firm.