Wading through the Muddy Waters of Observability, AIOps and All Things Monitoring and Performance
Observability is winning the buzzword wars, but what matters is the need to understand what's going on in enterprise environments and what to do about it. Terminology muddiness is making that harder.
💡 DXI Executive Insights are deeper, thought leadership research that analyzes the market (but without comparing vendors) and either analyzes one or more vendor’s competitive positioning within it or promotes a new perspective on it.
To paraphrase the diddy of Love Actually's aging rocker, Observability is All Around. The term has unquestionably won the buzzword wars — well, at least for now. But beneath this surface skirmish, there's a more fundamental truth: enterprise IT leaders are in a heated battle with their own environments. As complexity spins out of control, they are necessarily grasping for anything that will help them accomplish their two core needs: to understand what's going on within their environments, and figuring out what to do about it.
Thanks for reading The DX Report! Subscribe for free to receive new posts and support my work.
There is a long list of technologies that have sought to help enterprise IT leaders do just that, beginning with traditional infrastructure monitoring and application performance management tools and culminating with the more recent class of AIOps and Observability platforms. But the fights over terminology get in the way of helping enterprise IT leaders actually address their challenges.
Therefore, it's time to wade through these muddy waters and get down to the core of the issue — and plot a path forward.
The Troubled Path from IPM and APM to AIOps and Observability
We began working on this analysis several months ago with a simple goal of delving into the AIOps sector of the market. But it took about fifteen minutes to realize that it was a fool's errand. Everywhere we turned, vendors were using a whole array of terms to describe what they were doing — and there was little consistency in any of it.
The truth is that the current state is the result of a long march from the earliest days of this category in which we simply talked about monitoring. Starting with infrastructure and eventually moving on to application performance management (APM), the initial goal was to instrument the environment and establish the equivalent of tripwires to alert an IT organization when something was going off the rails.
However, as complexity and the volume of these alerts increased, this approach became increasingly untenable. AIOps tools emerged with the promise of using algorithms and eventually forms of AI to handle the resulting alert storms and identify the real potential issues from the false positives. Then, observability tools emerged from the premise that it was becoming impossible to identify every potential source of an issue (and, thereby, instrument it) in advance. The answer, these tools proclaim, is to collect log, metric, and trace data so that you can simply observe what is happening in the environment and identify issues from those observations.
Finally, vendors across this spectrum have begun to recognize that all of this data is a treasure trove that can help organizations not only react in the moment (reactively and proactively) to respond to issues, but also leverage that data for more strategic planning efforts.
Along the way, these terms got all jumbled about as tech companies attempt to solve various aspects of these challenges and leverage one term or another to describe what they are doing.
But here's the thing: all these nuanced terms have only served to muddy the water. And they fail miserably at describing what all of this means in the increasingly hybrid, cloud-native world of enterprise IT.
Observability in a Cloud-native, but Hybrid World
Much of the issue around these cascading terms is that they have been borne out of a need to articulate the shift occurring within enterprise IT environments.
The rise of cloud-native architectures and the increasing complexity of the enterprise tech stack created a dynamic in which traditional approaches to monitoring and performance management were too cumbersome to manage. Ultimately, this management difficulty resulted in visibility gaps.
These new terms signified new approaches. That was great, but ironically, they've also been inadequate in addressing the true nature of the complexity challenge in two regards.
First, observability and AIOps sound like a common sense approach to solving the complexity challenge. Collect all the data and let AI sort it out. The problem is that in true cloud-native environments, that data becomes a costly management challenge in its own right — meaning that the ability to gain visibility requires more than just new tooling, it requires new approaches and mindsets.
Second, traditional enterprises have not moved to cloud-native architectures in some wholesale, forklift manner. Their reality now — and for the foreseeable future — is a hybrid environment in which they have traditional legacy environments, traditional cloud-based environments, and an increasing presence of cloud-native architectures. And all of these are intermixed with mission critical business processes often functioning across all of them.
The net result is that there is no single answer — and definitely no single tool — to address the visibility and management challenges facing enterprise IT organizations. Instead, what's needed is a broader, curated approach.
A Broader View: Why You Need to Focus on Observability & Intelligence
The biggest challenge with the muddy waters of observability, AIOps and the rest of them is that the distinctions no longer serve or provide any benefit to an enterprise IT leader. Moreover, they've become almost meaningless.
Vendors have begun to use the terms (all of them) interchangeably and almost any vendor providing meaningful value is necessarily crossing the spectrum in terms of the solutions they're delivering to the market.
As we began to look at the market holistically, we quickly realized that all of these terms and approaches really came down to just two critical actions for enterprise IT leaders:
Understand what is happening within the enterprise tech stack, in context and in relationship to the other parts of the environment
Distill and transform that knowledge into actionable insights that enable them to protect service provision and create a strategic capability from that data
We are choosing to call this first action observability. We believe it most accurately describes the business objective: to observe and understand what is happening in the environment. But in our view, observability includes the entire process of data capture. Not only the collection of log, metric, and trace data, but traditional monitoring feeds and any other data which will help the organization get to this level of understanding.
We call the second action intelligence. Here again, our use of this term is intended to be broad, encompassing tactical intelligence for remedial action as well as strategic intelligence that informs planning, cloud migration, and transformational activities. Both are necessary and both should be derived from the underlying data collected as part of the observability process.
Then, within those two broad actions, you need to create the coverage applicable to your environment. This will almost certainly involve one or more of what we call the "Majors" that will often provide broad coverage and act as a focal point. You will then augment that will additional layers to deal with specific elements of your environment, such as a targeted observability tool for your cloud-native environments.
The critical factor, however, is that you must look at all of it as a unified and integrated whole. Allow the market to force you into looking at it in narrow, specific buckets and you'll doom yourself to silo hell and service failures.
Introducing our IT Observability & Intelligence Platform Sector Coverage
To help you take this broader view, we will focus our coverage on what we call the IT Observability & Intelligence Platform Sector. We will not make any attempt to distinguish between monitoring, observability, APM, AIOps or any other label. We will only look at this sector through the prism of a vendor's ability to help you create either understanding or intelligence around some element of your environment.
This is a very crowded sector with many niche players catering to specific elements of the tech stack, so expect our coverage to continue to evolve over time. We will be seeking out what we call Market Movers and offering their profiles over time. For now, here's how we're presently seeing the market.
Notes: If a vendor isn't listed below, it either means that they are not on our radar and we are not aware of them, or we have reviewed them and do not believe they are viable as an enterprise technology option. All vendors are listed alphabetically. Numerous Market Mover Profiles are currently under development and will be linked here upon completion.
These are the big players. You know who they are and can decide for yourself if they're a fit. We don't generally expect significant innovation here, but will cover it when we see it.
Market Movers are enterprise tech providers that we believe are delivering noteworthy, disruptive, and innovative solutions, potentially within a specific niche. They are worthy of consideration.
DRYiCE iControl (HCL)*
*Market Mover Profile pending
The Watch List
These are companies that are on-radar with us. We are either still evaluating them or waiting for a briefing, but we believe they are worth a look.
If you believe someone should be on our watch list, please email us at email@example.com
Thanks for reading The DX Report! Subscribe for free to receive new posts and support my work.