Karpenter versus the DevZero node autoscaler

The DevZero node autoscaler, also called dzkarp, is the provisioning and consolidation engine inside the DevZero Node Operator. It's built on open source Karpenter and runs on Amazon Elastic Kubernetes Service (Amazon EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE). It keeps Karpenter's APIs, so your NodePool and NodeClaim resources and your provider node classes (EC2NodeClass, AKSNodeClass, and GCENodeClass) work without changes.

On top of that base, dzkarp and the DevZero platform change which nodes get launched and when they get removed. New nodes use your unused Savings Plan commitment first. Spot types that get reclaimed less often rank higher. Consolidation accounts for how long your workloads take to restart, and pods can move to a new node with their in-memory state through live migration. This page explains each difference and how to configure it.

Who this page is for

This page is for platform and infrastructure engineers who run or are evaluating Karpenter and want to know what changes when they switch to DevZero. It assumes you know Karpenter concepts such as NodePools, consolidation, and disruption budgets. It doesn't cover installation. For setup steps, see the Node Operator overview.

Summary

The following table compares open source Karpenter with the DevZero node autoscaler. For features that DevZero is building, see In development.

Cloud support

Every dzkarp feature on this page works on Amazon EKS, AKS, and GKE, with two exceptions. Savings Plans awareness and the node price and pod IP capacity annotations depend on AWS billing and networking APIs, so they're available on Amazon EKS only.

The DevZero platform features later on this page, such as live migration, continuous resizing, scheduled overrides, and Simulate, run in the DevZero control plane and the Workload Operator, not in the node autoscaler.

Compatibility with Karpenter

dzkarp tracks upstream Karpenter releases and keeps the upstream APIs and their defaults. DevZero adds fields and resources in the devzero.karpenter.sh API group, and every addition is optional. If you already run Karpenter, see Migrate from Karpenter for a zero-downtime switch.

Savings Plans awareness

Open source Karpenter prices every on-demand instance at its list price. If you've bought EC2 Instance Savings Plans, Karpenter doesn't know about them, so it can launch an instance family you aren't committed to while your commitment goes unused.

On Amazon EKS, dzkarp reads your active EC2 Instance Savings Plans for the cluster's region and calculates how much hourly commitment each instance family has left. It then ranks capacity in this order, from first choice to last:

1. On-Demand Capacity Reservations (ODCRs)
2. On-demand instances in a family with unused Savings Plan commitment
3. Spot instances
4. On-demand instances at list price

To keep simultaneous launches from overspending the commitment, dzkarp reserves commitment for each launch request before it calls EC2.

Savings Plans tracking works as follows:

• Savings Plans data refreshes every 5 minutes.
• dzkarp assigns commitment to every running instance in the account and region, including instances that Karpenter didn't launch, and plans only against the commitment that's left.
• Compute Savings Plans apply to any instance family, so dzkarp doesn't need to steer launches toward them.
• The devzero.io/price-per-hour node annotation always shows the real billable rate.

The feature is always on and has no settings. The node autoscaler's IAM role needs savingsplans:DescribeSavingsPlans and savingsplans:DescribeSavingsPlanRates. The DevZero AWS CloudFormation template grants both.

Stable spot

Open source Karpenter picks spot capacity mostly by price. On AWS, EC2 Fleet's price-capacity-optimized strategy also favors pools with spare capacity at launch. On every cloud, Karpenter has no record of which instance types get reclaimed most often in your region.

dzkarp adds a stability ranking on Amazon EKS, AKS, and GKE. Every day, DevZero's control plane collects the spot interruption data that the cloud provider publishes for the cluster's region. The data measures the share of capacity reclaimed over the trailing 30 days. The control plane writes it to the cluster as a SpotStabilityData resource.

dzkarp sorts each spot type into one of five tiers and adds a penalty to the price it uses for ranking. The following table lists the tiers:

A stable instance type that costs slightly more wins over a cheaper type that gets reclaimed often, so your nodes see fewer interruptions and your pods restart less.

The penalty only affects ranking. It never removes an instance type from consideration, so spot launches are never blocked. If the stable types aren't available, dzkarp falls back to the normal Karpenter order. Cost reporting uses the real price, not the penalized one.

dzkarp turns off the ranking automatically when the data is older than 7 days or doesn't match the cluster's provider and region.

Stable spot is on by default and needs no setup. To turn it off, start the controller with --disable-spot-stability-pricing or set DISABLE_SPOT_STABILITY_PRICING=true.

Runtime settings

In open source Karpenter, feature gates and most tuning options are flags or environment variables. Changing one means editing the Helm release and restarting the controller.

dzkarp reads the same options from a cluster-scoped KarpenterSettings resource named default and applies changes without a restart. Helm values still provide the starting values. The resource's status shows the bootstrap values, the revision in effect, when it was applied, and the last error, if any.

The following example sets the log level, the batching windows, and three feature gates:

apiVersion: devzero.karpenter.sh/v1alpha1
kind: KarpenterSettings
metadata:
  name: default
spec:
  logLevel: info
  batch:
    maxDuration: 10s
    idleDuration: 1s
  featureGates:
    spotToSpotConsolidation: true
    nodeRepair: true
    cnpgAwareConsolidation: false

You can set these fields at runtime:

• logLevel
• batch.maxDuration and batch.idleDuration
• scheduling.preferencePolicy and scheduling.minValuesPolicy
• ignoreDRARequests and disableClusterStateObservability
• Feature gates: spotToSpotConsolidation, nodeRepair, nodeOverlay, reservedCapacity, staticCapacity, cnpgAwareConsolidation, capacityBuffer, terminateFirstDrift, terminateFirstRepair, and podDeletionCostManagement

Consolidation controls

Pod-level do-not-disrupt override

Many Helm charts add the karpenter.sh/do-not-disrupt: "true" annotation to their pods. In open source Karpenter, one such pod keeps its node from being consolidated, so large, mostly empty nodes can keep running.

dzkarp adds spec.disruption.ignorePodDoNotDisrupt to the NodePool. When it's true, consolidation, drift, and drain ignore the pod-level annotation on that pool. PodDisruptionBudgets and the node-level do-not-disrupt annotation still apply.

The following NodePool turns on the override:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    ignorePodDoNotDisrupt: true

The default is false. Changing the field doesn't drift existing nodes. dzkarp emits a PodDoNotDisruptIgnored event each time it overrides the annotation, so you can audit the decisions.

CloudNativePG-aware consolidation

A CloudNativePG primary is usually protected by a PodDisruptionBudget, so open source Karpenter doesn't consolidate the node it runs on.

With the cnpgAwareConsolidation feature gate on, dzkarp can treat that node as a consolidation candidate. It does this only when every primary on the node belongs to a healthy cluster with a ready replica on another node that isn't being disrupted. The CloudNativePG operator handles failover as usual. The karpenter_cnpg_candidates_unblocked_total metric counts how many nodes the gate made eligible.

The gate is off by default. Turn it on in KarpenterSettings:

apiVersion: devzero.karpenter.sh/v1alpha1
kind: KarpenterSettings
metadata:
  name: default
spec:
  featureGates:
    cnpgAwareConsolidation: true

Node visibility

dzkarp writes annotations to every node in the cluster, including nodes that Cluster Autoscaler or managed node pools create. DevZero uses them for cost reporting and for migrating node groups to NodePools, and you can read them with kubectl.

The following table lists examples of the metadata annotations on each cloud:

On Amazon EKS, dzkarp also writes the following price and capacity annotations:

The karpenter_node_capacity_pod_ipv4_capacity_max_pods_divergence metric flags nodes where the kubelet's max-pods value is higher than the number of IP addresses the CNI can assign. Pods on those nodes can get stuck in ContainerCreating.

Decision tracing

Open source Karpenter logs its decisions, but tying a pending pod to the node that was launched for it, or a terminated node to the disruption that caused it, means searching logs.

dzkarp stamps each decision with a correlation ID and emits a Kubernetes event at each lifecycle step:

• Annotations: karpenter.sh/provisioning-id, karpenter.sh/disruption-provisioning-id, karpenter.sh/disruption-termination-id, and karpenter.sh/spot-interrupt-termination-id
• Events: ProvisioningTriggered, ProvisioningLaunched, ProvisioningReady, DisruptionDecided, DisruptionReplacementLaunched, TerminationBegun, and TerminationSucceeded

Spot interruption events carry the interruption ID too. To trace a decision from start to finish, filter the output of kubectl get events by its ID.

Features from the DevZero platform

The preceding sections describe changes inside dzkarp. The DevZero control plane and the Workload Operator add the following features. They work together with dzkarp, and some of them, such as scheduled overrides and Simulate, also work on clusters that run open source Karpenter.

Startup-time-aware consolidation

Consolidation saves money by moving pods off underused nodes. Each move restarts those pods, though. For a service that takes 10 minutes to warm up, a small saving can cost a lot of capacity while the service restarts.

Open source Karpenter doesn't know how long your workloads take to start. DevZero measures each workload's time to ready, from p50 to p99.9, and uses it to tune consolidation for each NodePool:

• DevZero recommends the Balanced policy for churn-sensitive pools. A pool is churn sensitive if it runs workloads that take 5 minutes or more to restart at p95, or if 30% or more of its capacity has no PodDisruptionBudget or runs workloads with volatile CPU use. Balanced approves a move only when the savings outweigh the disruption.
• DevZero suggests a terminationGracePeriod based on p95 startup time when that time is over 30 seconds.
• DevZero suggests a consolidateAfter value from each pool's daily utilization troughs. The value is never below 10 minutes, and the suggestion only shortens the current wait.

Review and apply these suggestions on the node policy in the DevZero dashboard.

Scheduled overrides

Open source Karpenter supports schedules only for disruption budgets. The node requirements themselves are fixed until you edit the NodePool.

DevZero node policies support scheduled overrides. Each override is a time window that changes part of the policy while it's active. For example, you can allow only spot capacity on weekends, block large instance types overnight, or pause consolidation during a release window. The following table lists the fields of an override:

An override can change instance categories, families, sizes, generations, and types, as well as zones, architectures, capacity types, and operating systems. It can also change labels, taints, CPU and memory limits, and disruption settings such as consolidateAfter, expireAfter, budgets, and ignorePodDoNotDisrupt.

DevZero resolves the active policy every 60 seconds and records the active overrides on the NodePool in dakr.devzero.io/active-overrides. When a window ends, nodes return to the base policy through normal drift, paced by your disruption budgets. If the cluster loses its connection to DevZero, it keeps the last policy it received.

The node policy editor shows each schedule in plain language with a badge for the active or next window. A preview toggle shows the effective policy at the present time or at a time you pick.

Live migration

When open source Karpenter consolidates a node, or a spot node gets reclaimed, it evicts the pods and Kubernetes starts them again from scratch. The pods lose their in-memory state, caches, and warm-up progress.

DevZero can move those pods with live migration instead. The Workload Operator checkpoints the running process with Checkpoint/Restore In Userspace (CRIU), restores the checkpoint into the replacement pod on the target node, and keeps emptyDir contents. The application picks up where it stopped.

Live migration runs on node drains, pod evictions, preemption, spot interruptions, and resource changes. Deployments, StatefulSets, and DaemonSets use rolling checkpoints. For Pods, Jobs, CronJobs, ReplicaSets, Argo Rollouts, Volcano jobs, Spark applications, Kubeflow notebooks, and CloudNativePG clusters, the operator takes a checkpoint before it recreates them.

Before it migrates a pod, the operator checks that the running image matches the pod template. If a pod can't be migrated, it restarts normally.

To turn on live migration, do the following:

1. In the DevZero dashboard, turn on checkpointing for the node policy. DevZero labels the policy's nodes with dakr.devzero.io/checkpoint-node=true.
2. On the workload optimization policy, set live_migration_enabled.

The node group status changes to ENABLED after DevZero confirms that CRIU works on its nodes.

Continuous workload resizing

Karpenter picks nodes from pod requests. If the requests are too high, Karpenter launches nodes that sit mostly idle, and consolidation can't fix that.

DevZero resizes workloads continuously. Its multidimensional pod autoscaler sets each container's CPU and memory requests and limits and the replica count in one pass, based on observed usage over a lookback window (7 days by default). The in-cluster operator also reacts within seconds to out-of-memory (OOM) kills and sustained CPU throttling.

DevZero applies changes through in-place vertical scaling on Kubernetes 1.33 and later, through live migration, or through a normal rollout. As requests shrink, pods fit on fewer nodes, and dzkarp consolidates the nodes that are freed.

Set the targeting, triggers (manual, on a schedule, or on detection), CPU and memory rules, change threshold, and spike protection on a workload optimization policy.

Simulation and compaction

Karpenter simulates scheduling before each consolidation move. It can't show you how your cluster looks under a different NodePool before you apply one.

DevZero Simulate shows you that result first. Every node group on the DevZero node analysis page, including Karpenter NodePools, Cluster Autoscaler groups, and DevZero node policies, has a Simulate button. It packs the group's running pods onto the instance types that your edited constraints allow, counts DaemonSet overhead, and shows:

• Node count and monthly cost, before and after
• Risk flags such as pods that can't be placed, low instance diversity, tight headroom, architecture changes, a changed spot mix, and Reserved Instance or Savings Plan coverage that a family change leaves unused

You can also simulate removing specific nodes. Simulate is read-only and never changes the cluster.

Node compaction removes underused nodes on both Karpenter and Cluster Autoscaler clusters. In the dashboard, you preview the plan for a compaction policy and run it on demand. Your DevZero team can also turn on automatic planning, which plans removals every 15 minutes. Plans account for affinity, topology spread, taints, and zonal volumes. The operator cordons and drains each node, respects PodDisruptionBudgets, and never force-deletes pods. Compaction policies support ignore rules, an underutilization threshold, and maintenance windows.

Scheduling health

Open source Karpenter exposes Prometheus metrics, and you build the dashboards. The Scheduling health view on the DevZero node analysis page answers four questions for Karpenter and Cluster Autoscaler clusters. The following table describes each section:

In development

DevZero is building the following features. For timelines or early access, contact your DevZero team.

Next steps