Troubleshooting

Start with the logs. Most problems show up in the first lines after startup:

kubectl logs -n devzero-system deploy/trezr | head -50

Pod Is Running but No Cost Data Arrives

The logs show:

UNABLE TO RESOLVE CLUSTER TOKEN — running DEGRADED: probes answer, ...

Cause

trezr found no cluster token. It reads the token from the Read Operator's devzero-zxporter-token Secret, then the devzero-zxporter-env-config ConfigMap. With no token, trezr stays up so health checks pass, but it collects nothing. After 15 minutes it exits, and Kubernetes restarts it to look again.

Fix

1. Check the Read Operator is installed and running:

   kubectl get pods -A | grep zxporter

2. Check the Secret exists in trezr's namespace. trezr only reads the Secret from its own namespace:

   kubectl get secret devzero-zxporter-token -n devzero-system

3. If the Read Operator runs in another namespace, point the ConfigMap fallback at it:

   --set trezr.tokenFromConfigMap.namespace=<zxporter-namespace>

Once the token exists, trezr picks it up on its next restart.

Access Tier Is Minimal

The startup log shows:

permission probe complete	{"provider": "aws", "access_tier": "minimal"}

Cause

trezr's identity cannot list commitments. The most common reasons:

Fix

Check the identity is attached:

kubectl get sa trezr -n devzero-system -o jsonpath='{.metadata.annotations}'

Grant the missing role from the setup guide for your cloud (AWS, GCP, Azure). Then restart trezr. It checks its access only at startup.

kubectl rollout restart -n devzero-system deploy/trezr

Helm Install Fails: ServiceAccount Exists

On AWS, the install fails with an error that the trezr ServiceAccount exists and cannot be imported into the current release.

Cause

eksctl create iamserviceaccount created the ServiceAccount. Helm will not take over an object without its own ownership labels.

Fix

Add them, then install again:

kubectl label sa trezr -n devzero-system \
  app.kubernetes.io/managed-by=Helm --overwrite
kubectl annotate sa trezr -n devzero-system \
  meta.helm.sh/release-name=trezr \
  meta.helm.sh/release-namespace=devzero-system --overwrite

No Exports Listed Under Billing Data

The Billing data section in DevZero has no export to pick.

Cause and fix

1. trezr has not run discovery yet. It lists exports once at startup, then every 24 hours. If you created the export after trezr started, restart it.
2. The identity cannot list exports.
   • AWS: grant bcm-data-exports:ListExports and bcm-data-exports:GetExport. DevZero shows the error when this is the cause.
   • GCP: grant roles/bigquery.dataViewer on the export dataset. Also check trezr.gcp.bigqueryDataset names the right dataset, or leave it empty to scan the project.
   • Azure: grant Cost Management Reader on the subscription.
3. The export is the wrong kind.
   • AWS: the export must be a CUR 2.0 Data Export with the 23 columns trezr reads. An export missing columns is listed with the columns it lacks, and cannot be picked.
   • GCP: the table must be gcp_billing_export_resource_v1_*, the Detailed usage cost export. The standard usage cost export does not have per-resource rows.
   • Azure: the export must be Cost and usage details (FOCUS) with a Daily frequency.

Node Costs Stay Estimates After Picking an Export

Cause and fix

1. The export has no data yet. AWS delivers the first files up to 24 hours after you create the export. GCP takes a few hours. Azure writes once a day.
2. trezr cannot read the files.
   • AWS: the role needs s3:ListBucket and s3:GetObject on the export bucket. The default policy covers only devzero-trezr-cur-*. The bucket must also be in the cluster's own account.
   • GCP: the service account needs roles/bigquery.jobUser on trezr.gcp.bigqueryJobProject, which defaults to the billing project.
   • Azure: the identity needs Storage Blob Data Reader on the export's storage account.
3. The newest hours are always estimates. Clouds bill a day or more late. On GCP, CUD credits arrive up to 1.5 days late. Older days should show billed costs.

Look for the read in the logs:

kubectl logs -n devzero-system deploy/trezr | grep -E "ingesting billing month|using billing export table"

trezr Is OOM-Killed During a CUR Read

Large CUR files can push memory past the default 512 MiB limit. Raise it:

helm upgrade trezr oci://registry-1.docker.io/devzeroinc/trezr \
  --namespace devzero-system --reuse-values \
  --set resources.limits.memory=1Gi

If the pod is evicted for disk use instead, raise curScratch.sizeLimit from its 2 GiB default.

NAT Gateway Charges Are Missing (AWS)

Cause

trezr ties NAT gateway and VPC endpoint charges to a cluster through a snapshot of its VPCs. Without ec2:DescribeNetworkInterfaces, there is no snapshot, so these charges are never collected.

Fix

Add the five network actions from the IAM policy to trezr's role. The CLI and dz-installer grant them by default. Roles created before they were added may lack them.