Azure AKS

Set Environment Variables

First, set the following variables

export CLUSTER_NAME="<cluster-name>"
export RG=$(az aks list --query "[?name=='${CLUSTER_NAME}'].resourceGroup" -o tsv)
export LOCATION=$(az aks list --query "[?name=='${CLUSTER_NAME}'].location" -o tsv)

Setup dzKarp Auth

Create the workload MSI that backs the dzKarp pod auth:

KMSI_JSON=$(az identity create --name karpentermsi --resource-group "${RG}" --location "${LOCATION}")

Ensure Cluster has OIDC, workload and managed identity enabled, this may cause node disruption:

az aks update --name "${CLUSTER_NAME}" --resource-group "${RG}" \
  --enable-oidc-issuer --enable-workload-identity --enable-managed-identity

Create federated credential linked to the dzKarp service account for auth usage:

AKS_JSON=$(az aks show --name "${CLUSTER_NAME}" --resource-group "${RG}")
az identity federated-credential create --name KARPENTER_FID --identity-name karpentermsi --resource-group "${RG}" \
  --issuer "$(jq -r ".oidcIssuerProfile.issuerUrl" <<< "$AKS_JSON")" \
  --subject system:serviceaccount:devzero-system:karpenter-sa \
  --audience api://AzureADTokenExchange

Create role assignments to let dzKarp manage VMs and Network resources:

KARPENTER_USER_ASSIGNED_CLIENT_ID=$(jq -r '.principalId' <<< "$KMSI_JSON")
RG_MC=$(jq -r ".nodeResourceGroup" <<< "$AKS_JSON")
RG_MC_RES=$(az group show --name "${RG_MC}" --query "id" -otsv)
for role in "Virtual Machine Contributor" "Network Contributor" "Managed Identity Operator"; do
  az role assignment create --assignee "${KARPENTER_USER_ASSIGNED_CLIENT_ID}" --scope "${RG_MC_RES}" --role "$role"
done

Configure helm values

dzKarp for aks helm values can be configured with a helper script.

export DZKARP_VERSION=1.7.7
# get the valus template
curl -sO https://raw.githubusercontent.com/devzero-inc/dakr-operator-installers/refs/tags/${DZKARP_VERSION}/dzKarp/azure/karpenter-values-template.yaml
# get the helper script
curl -sO https://raw.githubusercontent.com/devzero-inc/dakr-operator-installers/refs/tags/${DZKARP_VERSION}/dzKarp/azure/configure-values.sh
# run the helper script
chmod +x ./configure-values.sh && ./configure-values.sh ${CLUSTER_NAME} ${RG} karpenter-sa karpentermsi

Install dzKarp

Install dzKarp with helm:

helm upgrade --install karpenter oci://public.ecr.aws/devzeroinc/dzkarp-azure/karpenter \
  --version "${DZKARP_VERSION}" \
  --namespace devzero-system --create-namespace \
  --values karpenter-values.yaml \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --set settings.implicitPDBMinAvailable="20%" \
  --set settings.featureGates.spotToSpotConsolidation=true \
  --wait

implicitPDBMinAvailable specifies the minimal available subset of pods in any deployment or statefulset dzkarp must maintain during node disruption for which an explicit pdb is not already defined. This value can be a percentage string or a natural number (for a nominal pod count). Set this value to zero if you do not want a implicit fallback pdb.

Verify dzKarp

kubectl logs -f -n devzero-system -l app.kubernetes.io/name=karpenter -c controller

see that no unexpected errors are produced

Set nodeAffinity for critical workloads (optional)

Autoscaled nodes can be prone to churn and result in workload disturbance.

You may want to set a nodeAffinity critical cluster workloads to mitigate this.

Some examples are

• coredns
• metric-server

add the following to your cluster critical workload deployments

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: karpenter.sh/nodepool
          operator: DoesNotExist

Create Node Policy

We need to create a Node Policy in DevZero, and have it target the cluster on which dzKarp was just installed.

Head over to the optimization dashboard, click on "Create Node Policy" and follow the form to create a policy suitable for your needs.

After the Policy is created, click on it in the menu and point it at the cluster you just created via "Create Target".

In about a minute this should create nodepool and nodeclass objects in your kubernetes cluster.

Check them out:

kubectl describe aksnodeclass
kubectl describe nodepools

Migrate workloads onto autoscaled nodes

To get rid of the instances that were added from the node pool we can scale our node pools down to a minimum size to support dzKarp and other critical services.

If you have a single node pool, we suggest scaling it to 2 instances:

az aks nodepool scale \
    --cluster-name "${CLUSTER_NAME}" \
    --resource-group "${RG}" \
    --name <your-node-pool-name> \
    --node-count 2

Or, if you have multiple node pools, choose one to keep and delete the others.

Workloads that select on agentpool

AKS labels every node in a node pool with agentpool=<pool-name>. dzKarp nodes do not get this label, so a workload pinned to a pool, for example with nodeSelector: {agentpool: <pool-name>}, stays Pending instead of moving onto dzKarp nodes.

To move such a workload without changing its manifest, add the label agentpool: <pool-name> to the Labels of the Node Policy that should serve it (requires dzKarp 1.9.4 or later). dzKarp then puts that label on every node it creates for the policy. Policies generated from your existing node groups don't copy agentpool, so add it yourself.

If you use the agentpool label on dzKarp nodes:

1. Add the label to the Node Policy.
2. Delete the AKS node pool with that name (az aks nodepool delete). Its pods move onto dzKarp nodes carrying the label. If dzKarp has already created nodes with the label, they are replaced once as part of this step.
3. Never create a new AKS node pool with that name. Deleting it later would remove the dzKarp nodes again.

Do not leave the pool in place at zero nodes as a substitute for deleting it. Whoever deletes the empty pool later also removes your dzKarp nodes.

If you can change the workload instead, select dzKarp nodes by the label karpenter.sh/nodepool: <nodepool-name> (the NodePool your Node Policy created, see kubectl get nodepools), or by a label of your own without the kubernetes.azure.com prefix. This avoids the coupling with AKS node pool names entirely.

As node pool nodes are drained you can verify that dzKarp is creating nodes for your workloads.

kubectl logs -f -n devzero-system -l app.kubernetes.io/name=karpenter -c controller

You should also see new nodes created in your cluster as the old nodes are removed.

kubectl get nodes