Kratix Ep 10: Observability and Operations

1. The Multi-Cluster Observability Problem

#

In a mature Kratix architecture, failures can happen in three entirely distinct geographic or logical locations:

1. The Pipeline (Platform Cluster): An E-Commerce developer submitted a bad Claim, and your Bash script crashed during processing.
2. The Delivery (GitOps): ArgoCD on the prod-us-east Worker cluster failed to pull from Git due to GitHub network issues or expired SSH keys.
3. The Execution (Worker Cluster): The physical Redis pod crashed due to an Out Of Memory (OOM) error.

To troubleshoot effectively, a Platform Engineer must be able to see all three of these metrics on a single Grafana dashboard simultaneously.

2. Architecting the Monitoring Stack

#

The undisputed industry standard for Kubernetes monitoring is the kube-prometheus-stack (which bundles Prometheus, Grafana, and Alertmanager).

To monitor a Kratix fleet, we must implement a Hub-and-Spoke metrics model:

• Worker Clusters: We will run a lightweight Prometheus agent on every Worker Cluster. Its only job is to scrape local pod metrics (like Redis CPU usage or ArgoCD sync status) and push them via the remote_write protocol to the central Hub.
• Platform Cluster (The Hub): We will run a massive, persistent Prometheus Server here to store all the incoming data, and host the Grafana Web UI.

3. Configuring the Platform Cluster (The Hub)

#

First, we must install the central observability stack on the Platform Cluster and configure it to accept incoming metrics.

# Add the Prometheus community helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install the stack in the Platform Cluster
helm install central-monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.enableRemoteWriteReceiver=true

The enableRemoteWriteReceiver=true flag is absolutely critical. By default, Prometheus only pulls data. This flag opens an HTTP endpoint that allows the Worker clusters to push data inward.

4. Configuring the Worker Clusters (The Agents)

#

Now, switch your kubectl context to a Worker Cluster (e.g., prod-us-east).

We will install a lightweight Prometheus instance. We configure it to scrape everything locally, but we configure its remoteWrite destination to point to the Platform Cluster’s Ingress URL.

Create a values-worker.yaml file:

prometheus:
  prometheusSpec:
    # Send all scraped data to the Platform Cluster!
    remoteWrite:
      - url: "https://prometheus.platform.acmecorp.com/api/v1/write"
    
    # Optional but highly recommended: Add a label so you know which cluster sent the data
    externalLabels:
      cluster_name: "worker-prod-us-east"
      
# We don't need Grafana on the worker clusters, the developers will view the central one
grafana:
  enabled: false

Install it on the Worker Cluster:

helm install worker-monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f values-worker.yaml

5. Building the Kratix Grafana Dashboard

#

Now that data is flowing continuously from the Spokes to the Hub, you can log into Grafana on the Platform Cluster.

You can create a beautiful custom dashboard that tracks the lifecycle of a Kratix Promise end-to-end. Here are three critical PromQL queries you should include:

Panel 1: Pipeline Success Rate

#

Query the native Kratix controller metrics (running locally in the Platform Cluster) to see how many Pipelines are failing globally.

rate(kratix_pipeline_executions_total{status="error"}[5m])

Panel 2: GitOps Sync Status

#

Query the ArgoCD metrics (originating remotely from the Worker clusters) to ensure all State Destinations are syncing perfectly across the globe.

argocd_app_sync_status{status="OutOfSync"}

Panel 3: Workload Health

#

Query standard kube-state-metrics (also originating remotely from the Worker clusters) to ensure the physical databases requested by developers are actually running and healthy.

sum by (cluster_name) (kube_pod_status_phase{phase="Failed"})

6. Automating Observability with Kratix

#

Here is the ultimate Platform Engineering flex: You shouldn’t install the Worker Monitoring stack manually.

You should create an ObservabilityPromise in Kratix!

The Promise would contain a Pipeline that generates the Helm YAML for the Worker Prometheus agent. When you spin up a brand new Worker Cluster in a new region, you simply apply an ObservabilityClaim to the Platform cluster. Kratix automatically deploys the monitoring agent to the new Worker via GitOps, instantly hooking the new cluster up to the central Grafana dashboard without any manual configuration.

This is the true power of treating your “Platform as a Product.”

Conclusion & Next Steps

#

You have successfully centralized the telemetry of a massively distributed system. By aggregating logs and metrics into the Platform Cluster, your Operations team can monitor the health of 100 worker clusters without ever leaving their primary Grafana dashboard.

We have now covered every individual component of a modern IDP: Crossplane, Kratix, ArgoCD, HashiCorp Vault, and Prometheus.

In Episode 11: Full IDP Integration with Unified Dashboards, we will pull back and look at the big picture. We will integrate this entire stack into Spotify’s Backstage, creating a single, beautiful Web UI where developers can request infrastructure, view deployment status, and see their Grafana metrics all in one place.