KCNA Ep 4: Telemetry, Observability & Monitoring

TL;DR (Quick Summary)

#

• The 3 Pillars:
  1. Metrics: Numeric aggregations over time (CPU usage, HTTP request rate).
  2. Logs: Timestamped text records of discrete events (stdout/stderr).
  3. Traces: End-to-end request journeys across microservice boundaries.
• Prometheus: Pull-based metrics monitoring system using PromQL.
• OpenTelemetry (OTel): Vendor-neutral CNCF framework for collecting, generating, and exporting telemetry data.
• Jaeger: Distributed tracing system for pinpointing latency bottlenecks.

1. The Three Pillars of Observability

#

graph TD
    subgraph Observability ["The Three Pillars"]
        M["1. Metrics
(Prometheus)"]
        L["2. Logs
(Fluentbit / Loki)"]
        T["3. Traces
(Jaeger / OpenTelemetry)"]
    end

    M -->|Answers| A1["'Is the system degraded?'
(CPU 95%, Error Rate > 5%)"]
    L -->|Answers| A2["'Why did it fail?'
(NullPointerException in line 42)"]
    T -->|Answers| A3["'Where is the latency?'
(DB Query took 2.4s out of 2.6s)"]

2. Prometheus Metrics Architecture

#

Prometheus is a Graduated CNCF project that monitors systems by pulling (scraping) HTTP metrics endpoints exposed by applications in the OpenMetrics format.

graph LR
    AppPod["Application Pod
(Exposes /metrics)"] <-- Scrapes metrics via HTTP -- PrometheusServer["Prometheus Server
(TSDB Engine)"]
    Exporter["Node Exporter
(Hardware Metrics)"] <-- Scrapes metrics -- PrometheusServer
    PrometheusServer -->|Alert Rules| Alertmanager["Alertmanager"]
    PrometheusServer -->|PromQL Queries| Grafana["Grafana Dashboard"]

Prometheus Data Model & PromQL

#

• Metric Types:

  1. Counter: Monotonically increasing number (e.g., total HTTP requests http_requests_total). Reset to 0 on restart.
  2. Gauge: Single numerical value that goes up and down (e.g., memory usage node_memory_active_bytes, CPU temperature).
  3. Histogram: Samples observations and counts them in configurable buckets (e.g., HTTP request durations).
  4. Summary: Similar to histogram, calculates configurable quantiles over a sliding time window.

• PromQL Example: Calculate 5-minute per-second rate of HTTP errors:

  rate(http_requests_total{status=~"5.."}[5m])

3. OpenTelemetry (OTel) & Distributed Tracing

#

OpenTelemetry is an Incubating CNCF project formed by merging OpenTracing and OpenCensus. It provides a standardized, vendor-neutral collection layer.

graph LR
    MicroserviceA["Microservice A
(OTel SDK)"] -->|OTLP Protocol| OTelCollector["OpenTelemetry Collector"]
    MicroserviceB["Microservice B
(OTel SDK)"] -->|OTLP Protocol| OTelCollector
    OTelCollector -->|Export Metrics| Prometheus["Prometheus"]
    OTelCollector -->|Export Traces| Jaeger["Jaeger"]

Distributed Tracing Terminology (Jaeger & OTel)

#

• Trace: Represents the entire journey of a single user request through a distributed system of microservices.
• Span: A single named, timed block of work within a trace (e.g., executing an SQL query, calling an external payment gateway).
• Trace Context Propagation: HTTP headers (traceparent, tracestate) passed between microservices to link individual spans into a single unified trace graph.

4. Key KCNA Exam Practice Questions

#

Question 1

#

Which Prometheus metric type represents a value that can arbitrarily go up and down, such as available system memory or active connection count?

• A) Counter
• B) Gauge (Correct)
• C) Histogram
• D) Summary

Rationale: Gauges measure values that fluctuate up and down. Counters can only increase or reset to zero.

Question 2

#

What CNCF project provides a unified, vendor-neutral specification and collector for metrics, logs, and traces?

• A) Prometheus
• B) Jaeger
• C) OpenTelemetry (Correct)
• D) Fluentd

Rationale: OpenTelemetry is the official CNCF standard framework for collecting and exporting all three telemetry signals.

Summary & Next Steps

#

In this episode, we covered:

• The Three Pillars of Observability (Metrics, Logs, Traces).
• Prometheus architecture, scrape model, and metric types (Counter, Gauge, Histogram).
• OpenTelemetry Collector and Jaeger distributed tracing concepts.

In KCNA Episode 5: Cloud Native Security & GitOps Practices, we will examine RBAC, NetworkPolicies, GitOps (ArgoCD/Flux), and Service Meshes!