Kratix Ep 7: Secrets Management with ESO and Vault

1. The Secrets Architecture (WHAT & WHY)

#

To solve the fundamental GitOps secret problem for our E-Commerce Redis database, we must introduce two new, industry-standard components to our Internal Developer Platform:

1. HashiCorp Vault: A centralized, heavily encrypted vault for storing secrets. (This is often hosted externally, outside of Kubernetes entirely, or managed via a SaaS offering like HCP Vault).
2. External Secrets Operator (ESO): A Kubernetes controller running natively on the Worker Cluster. Its sole purpose is to authenticate with external APIs (like Vault, AWS Secrets Manager, or Azure Key Vault), pull secrets down, and inject them securely into Kubernetes memory as standard Secret objects.

The Secure Data Flow (HOW)

#

To understand how these pieces fit together, let’s trace the lifecycle of a password:

1. Generation: The Kratix Pipeline (running in the Platform Cluster) generates a random, cryptographically secure 32-character password for the new Redis Database.
2. Evasion: The Pipeline does not write this password to the /kratix/output/ directory (because anything placed in that directory gets committed to Git).
3. The Side-Channel: Instead, the bash script inside the Pipeline makes a direct REST API call over HTTPS to HashiCorp Vault to store the password securely in the Vault backend.
4. The Pointer: The Pipeline then generates an ExternalSecret YAML manifest and writes that file to /kratix/output/. (An ExternalSecret is just a pointer or a map; it contains no sensitive data, only instructions on where to find the data in Vault).
5. GitOps Sync: Kratix commits the harmless pointer to Git. ArgoCD pulls the pointer down to the Worker Cluster.
6. Retrieval: ESO (running on the Worker Cluster) reads the ExternalSecret, authenticates with Vault, downloads the actual password securely into memory, and creates a standard Kubernetes Secret for the Redis pod to mount.

Throughout this entire process, the Git repository remains completely clean and free of sensitive data.

2. Writing the Secure Pipeline (HOW)

#

Let’s look at the actual bash script running inside our Kratix Pipeline container to see how this is executed programmatically.

We need the script to: generate a password, push it to Vault, and output an ExternalSecret for ArgoCD to deploy.

#!/usr/bin/env bash
set -e

# 1. Read the Developer's Claim from Kratix Input
CLAIM_NAME=$(jq -r '.metadata.name' /kratix/input/object.yaml)

# 2. Generate a random secure password for Redis
REDIS_PASSWORD=$(openssl rand -base64 24)

# 3. Push the password directly to HashiCorp Vault (The Side-Channel!)
# (Note: We assume VAULT_ADDR and VAULT_TOKEN are injected securely as environment 
# variables into the Pipeline pod by the Kratix controller).
curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data "{\"data\": {\"password\": \"$REDIS_PASSWORD\"}}" \
     $VAULT_ADDR/v1/secret/data/kratix/$CLAIM_NAME/redis-credentials

# 4. Generate the ExternalSecret Pointer YAML for Git
cat <<EOF > /kratix/output/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: $CLAIM_NAME-redis-secret
spec:
  # How often ESO should check Vault for updates
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    # This is the name of the standard K8s Secret that ESO will create!
    name: $CLAIM_NAME-redis-secret 
  data:
    - secretKey: password
      remoteRef:
        # Point to the exact path we just created in Vault via curl!
        key: secret/data/kratix/$CLAIM_NAME/redis-credentials
        property: password
EOF

3. Configuring the Worker Cluster (WHERE)

#

For the pointer to work, the Worker Cluster must have the External Secrets Operator installed and configured with credentials to talk to Vault.

Switch your kubectl context to your Worker Cluster.

Step 1: Install ESO

#

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
    -n external-secrets --create-namespace

Step 2: Configure the ClusterSecretStore

#

You must tell ESO how to log into Vault so it can fetch the passwords. We do this using a ClusterSecretStore.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.acmecorp.com:8200"
      path: "secret"
      version: "v2"
      auth:
        # In a production environment, Kubernetes Service Account Auth (JWT) 
        # is preferred, but for simplicity, we are referencing a static token here.
        tokenSecretRef:
          name: vault-token
          key: token

4. The Final Verification (WHEN)

#

When the Kratix Pipeline finishes its work, ArgoCD pulls the harmless ExternalSecret pointer down to the Worker Cluster.

ESO instantly detects the new custom resource, reaches out to HashiCorp Vault using the credentials in the ClusterSecretStore, retrieves the password, and creates a standard Kubernetes Secret.

Run this command on your Worker Cluster to verify the entire flow worked:

kubectl get secret <claim-name>-redis-secret -o yaml

You will see the base64 encoded password perfectly injected into the cluster, completely ready to be mounted as an environment variable by the Redis StatefulSet.

Conclusion & Next Steps

#

You have successfully decoupled sensitive secrets from your infrastructure state. By utilizing HashiCorp Vault as a secure side-channel, your Kratix pipelines can dynamically generate API keys, TLS certificates, database passwords, and cryptographic hashes without ever compromising the integrity of your GitOps repository or failing compliance audits.

Up until this point in the series, we have focused entirely on the Backend (Pipelines, Vault, ArgoCD, Crossplane) and the role of the Platform Engineer (WHO). But what does the end-user—the Application Developer—actually experience? Is this system truly easier for them?

In Episode 8: Developer Self-Service Workflow, we will step into the shoes of the E-Commerce Developer. We will look at how they discover the Redis Promise, how they author their Claims, and how they interact with the Platform Cluster on a daily basis to manage their environments.