This post recaps a session I co-presented, alongside Peter Han and Tyler Bonilla , at the super{summit} conference in August 2022.
It’s amusing to see how our industry flocks around the latest tech buzzword. It’s cool, hip and trending, so why not get on the bandwagon?
The new buzzword kid on the blocks is o11y - a.k.a. Observability!
Actually, observability is a topic very near and dear to my heart over the last 15 years of my software development journey. In this post, I'll discuss what observability is, why it is important, a few challenges, and how to overcome them using Metrics, Events, Logs, and Traces.
Reading along should get you abreast with some of the concepts, patterns, tools and cultural aspects of embracing observability to the best of my knowledge. All references to open source and vendor tools are for illustration purposes only to guide you on the path of embracing observability. Depending on the cost and maintenance overhead tradeoffs you could evaluate the right tool that suits your business needs.
In control theory , Observability lets us understand a system from the outside, by letting us ask questions, without knowing the inner workings of that system. Observability lets us deal with unknown unknowns .
In software parlance , Observability is the ability to ask new questions of the health of your running services without deploying new code
In a world of cloud-native loosely coupled services, polyglot persistence , and dynamic infrastructure , traditional metrics-based monitoring approaches are woefully inadequate when it comes to understanding system state, and triaging and diagnosing behavioural and performance issues.
Over the years, Kubernetes has become a developer’s choice for designing and deploying scalable and distributed applications. However, Kubernetes is unaware of the internal state of an application. However, its dynamic nature also gave rise to an increased number of problems for platform engineers who needed to keep track of its performance despite the pace.
These two are often used interchangeably; however, they are different in their application. By monitoring, you can verify if the infrastructure and applications are functioning as expected . On the other hand, observability provides you with comprehensive actionable insights to take steps towards improving performance and making the applications and the entire infrastructure more stable and resilient.
Managing networking in a monolithic architecture is a relatively simple task: the path between the client to the server is generally through a finite collection of points.
In a distributed microservice architecture , the network becomes much more critical and complex : the path between client and application got much more winding and harder to reason about due to the
It makes root cause analysis and incident resolution potentially a lot harder, yet the same questions still need answering:
In the past few years, much has been talked about and written about the “three pillars of observability”: metrics, logs, and traces .
However, as pointed out by Charity Majors, these 3 pillars are not finite and can be complemented with other pillars viz. Events and Profiles
Events can be analogous to structured logs, however, they differ from logs due to the below traits
Profiling is the act of measuring a program’s behavior using data we gather as our code executes (for example, frequency and duration of function calls, CPU time or memory usage, and more).
Profiling is a new addition to the Observability stack and the profiling SIG (special interest group) has just kickstarted. I will be keeping a close eye on it as this evolves and gets supported by various open source and vendor tools.
Below are some of the best practices for implementing Observability in your stack. They can be used to build common shim layers or shared libraries for crosscutting concerns . Developing such generic layers in your programming language of choice will avoid code duplication and help in applying standard conventions across the stack and simplify maintenance and upgrades .
Ingest, index and visualize all the logs from various sources in a centralized store using either open source (FluentBit, Loki, ELK..) or vendor tools (DataDog, Honeycomb, NewRelic)
Use JSON structured logging as an alternative to traditional logging. Logs written in JSON are easily readable by both humans and machines, and structured JSON logs are easily tabularized to enable filtering and queries.
Include meaningful information about the event that triggered the log, as well as the additional context that can help understand what happened, find correlations with other events, and diagnose potential issues that require further investigation. A few examples of fields are
— User Request Identifiers (X-Request-Id)
— Unique Identifiers (X-User-Id, X-Tenant-Id)
Avoid logging sensitive data and personally identifiable information (PII) that may be covered by data privacy and security regulations or standards like the European GDPR, HIPAA, or PCI DSS.
Use interceptors wherever applicable for logging gRPC/REST requests/responses and prefer using language constructs (MDC in Java or Go Context) to inject context across all logs that are part of the same request/response lifecycle.
Use appropriate log levels (INFO, WARN, ERROR) to avoid logging non-essential information that doesn’t help with diagnostics or root cause analysis resulting in increased time-to-insights, data volumes, and higher costs.
Set different retention policies (S3 IA, Glacier) for different types of logs, depending on the cost and compliance needs.
More metrics are always better if you have the right tools. Hence, gather all the infrastructure golden signals using either open source (Prometheus with Thanos or Cortex) or vendor tools (DataDog, Honeycomb..).
Build language-specific libraries (Micrometer, Prometheus) to instrument application code to emit key business metrics that need alerting (count of custom jobs failed, count of in-flight messages in the queue).
Depending on the use case, use the appropriate metric type to capture data points which can then be aggregated to build Sums, Gauges, Histograms
Like logging, use interceptors wherever applicable to capture gRPC/REST request/response metrics and prefer using language constructs (MDC in Java or Go Context) to inject context across all metrics that are part of the same request/response lifecycle.
RED (Rate, Error, Duration)
Request-scoped — For every request, check utilization, saturation, and errors.
USE (Utilization, Saturation, Error)
Resource-scoped — For every resource, check utilization, saturation, and errors.
Once the golden signals are collected, they can be used collectively for alerting, troubleshooting or tuning and capacity planning.
Cardinality is the number of unique combinations of metric names and dimension values . Choose which dimensions you want to attach to your metrics based on what meaningful information you want to extract from your telemetry data. Immutable infrastructures on Kubernetes and Containers lead to cardinality explosion as once a resource is created, it is never updated.
In addition to the default W3C Span Context , code can be instrumented to use custom key-value attributes to annotate a Span to carry information about the operation it is tracking.
For example, if a span tracks an operation that adds an item to a user’s shopping cart in an eCommerce system, you can capture the user’s ID, the ID of the item to add to the cart, and the cart ID.
An application can be instrumented for emitting traces either automatically or manually.
The monitoring market used to be dominated by proprietary vendors. Each vendor had its own share of pros and cons related to cost and feature support. In response, various free and open-source software projects started or were spun out of tech companies. Early examples include Prometheus for metrics and Zipkin and Jaeger for tracing. In the logging space, the “ELK stack” ( Elasticsearch , Logstash , and Kibana ) gained market share and became popular.
The market has hit an inflection point, and cloud-native architectures are much larger in scale, more distributed and too interdependent. Developers need the flexibility to choose and control the data they collect and analyze.
One key milestone was the merger of the OpenTracing and OpenCensus projects to form OpenTelemetry , a major project within CNCF.
The configuration below should help you get started locally by running Docker containers of Prometheus , Jaeger and OpenTelemetry Collector.
You can then instrument your applications either automatically by using the OpenTelemetry Agents or by manually instrumenting them using OpenTelemetry SDK.
In both approaches, metrics and traces from your applications will be sent to the OpenTelemetry collector either using gRPC or HTTP . OpenTelemetry collector can then be configured to build pipelines for filtering or enriching telemetry data before emitting them to the tools like Jaeger, Prometheus, DataDog or NewRelic.
This multiplexing gateway approach avoids first having to select the tool and then instrument your application using the selected tool’s SDKs or Agents. You can seamlessly swap out tools without changing your application code.
Similar to DevOps, one cannot buy observability off the shelf. Tooling is part of the equation — you’ll need a platform that can ingest, correlate and analyze data — but tools alone aren’t the key to observability. It’s more than deploying certain tools or adopting certain workflows since it has to be embraced in your engineering culture . The culture has to be supported and backed by the engineering leadership and applied to all aspects of the software development lifecycle.
Across the board, super{set} companies are embracing observability by leveraging some of the above patterns and tools as part of their engineering culture and using it to improve the developer and customer experience.
Are you passionate about Platform Engineering, or interested in working in the fast pace of a startup? Check out careers.superset.com for roles across the super{set] portfolio.
Subscribe for sharp takes on innovation, markets, and the forces shaping our future.
We're heads down building & growing. Learn what's new and our latest updates.