Crossplane Ep 9: Troubleshooting Sync Errors

1. The Troubleshooting Cascade

#

When a developer complains: “My database isn’t provisioning”, you must trace the request down the stack.

The architecture is layered: Claim -> XR -> MR -> Provider -> AWS API.

You start at the top and work your way down until you find the failure.

2. Step 1: Inspect the Claim

#

The Application Developer submitted a Claim (e.g., PostgreSQLInstance) into their namespace.

Run:

kubectl get postgresqlinstance -n <namespace>

What to look for: Look at the READY and SYNCED columns.

• If they are missing entirely, the XRD is not installed.
• If SYNCED is False, the developer made a syntax error in their YAML that violates the OpenAPI schema.

Run:

kubectl describe postgresqlinstance app-db -n <namespace>

Scroll to the bottom of the output and look at the Events: section. This is the most important debugging area in all of Kubernetes. If Crossplane rejected the YAML, it will explicitly state why here (e.g., ValidationError: storageGB must be an integer).

3. Step 2: Inspect the XR and Composition

#

If the Claim looks fine, we move one layer deeper to the Cluster-Scoped XR.

kubectl get xpostgresqlinstances

Run a describe on the XR:

kubectl describe xpostgresqlinstances <xr-name>

Look at the Events: section. Common Errors at this layer:

1. composition is not selected: The XR cannot find a Composition that matches its compositeTypeRef or Labels. Check your Composition YAML.
2. cannot render composed resource: The Composition was found, but the Patcher crashed. This usually happens if you tried to patch an integer into a string field without a Transform, or if a Required EnvironmentConfig was missing.

4. Step 3: Inspect the Managed Resource (MR)

#

If the Composition succeeded, it generated physical Managed Resources. We need to check if the AWS Provider successfully applied them.

First, find out what MRs exist in the cluster:

# This command lists EVERY Managed Resource currently in the cluster
kubectl get managed

Find the one that is failing (READY: False or SYNCED: False). Let’s assume it’s the Instance.rds.aws.upbound.io.

Run a describe on the specific MR:

kubectl describe instance.rds <mr-name>

Look at the Events: section. Common Errors at this layer:

1. cannot update resource: AccessDenied: The Provider pod reached AWS, but the IAM credentials configured in your ProviderConfig (from Episode 2) do not have rds:CreateDBInstance permissions.
2. cannot update resource: InvalidParameterValue: You patched a value that AWS rejected (e.g., you requested instanceClass: db.t3.micro but that instance type is not available in eu-central-1).

5. Step 4: Inspect the Provider Logs

#

If the MR Events are empty, or if they show cannot connect to Provider, the AWS Provider pod itself might be crashing.

This happens if the cluster runs out of memory, or if the Provider cannot reach the internet to download the AWS SDK.

Find the Provider pod:

kubectl get pods -n crossplane-system

Look for the pod prefixed with provider-aws-rds-xxxxx.

Read its logs:

kubectl logs provider-aws-rds-xxxxx -n crossplane-system

You are now looking at the raw Go execution logs of the Crossplane Engine. If there is a fatal authentication error, network timeout, or panic, it will be printed here.

6. The Crossplane CLI Trace Command

#

Manually digging through Claims, XRs, and MRs using kubectl describe is tedious.

The Upbound team built a dedicated command into the Crossplane CLI to automate this exact workflow. If you installed the CLI in Episode 1, you can run the trace command.

kubectl crossplane trace postgresqlinstance app-db -n default

Expected Terminal Output:

NAME                                         SYNCED   READY   STATUS
PostgreSQLInstance/app-db                    True     False   Waiting: composed resources are not ready
└─ XPostgreSQLInstance/app-db-x5g2p          True     False   Waiting: composed resources are not ready
   ├─ Instance.rds.aws.upbound.io/db-123     False    False   ReconcileError: AccessDenied to AWS API
   └─ SubnetGroup.rds.aws.upbound.io/sg-456  True     True    Available

The trace command instantly builds a visual tree of the entire dependency graph, highlighting exactly which MR is causing the failure and printing the exact AWS error message. This is the most powerful debugging tool in Platform Engineering.

Conclusion & Next Steps

#

You now possess the knowledge to debug any distributed infrastructure failure. By systematically checking the Claim, the XR, the MR, and the Provider, you can identify whether a bug is caused by a Developer typo, a Platform engineering patch error, or an AWS IAM misconfiguration.

This concludes the Intermediate Tier! You have mastered YAML-based Compositions.

However, as your architectures become more complex (e.g., iterating over arrays, calling external APIs, writing complex conditional logic), YAML patches become incredibly cumbersome. YAML was never designed to be a programming language.

In Episode 10: Introduction to Composition Functions, we enter the Advanced tier. We will learn how Crossplane is completely replacing YAML patches with Turing-complete programming languages, allowing you to write your Compositions in Go or Python.