Upgrade Kubernetes on OCI Clusters

This topic describes how to upgrade Kubernetes on OCI clusters with Smart Karpenter-managed nodes.

info

Kubernetes upgrades are supported only with Smart Karpenter version 1.2.0 or later.

Prerequisites

Verify Karpenter Compatibility

Confirm your Karpenter version supports the target Kubernetes version by referring to the Karpenter compatibility matrix. Upgrade Karpenter first if required.

note

Karpenter core is now v1.8.0. Before deploying Smart Karpenter v1.2.0, upgrade the Karpenter CRDs, then update the Helm chart or manifests. Deploying the new controller against stale CRDs will fail.

Verify Image Selection Mode

Check your NodePool and OciNodeClass image selection. There are two supported modes.

Option A - Implicit Image Selection

When imageOcid is not specified on the OciNodeClass:

• The image is derived automatically (for example, from existing OKE node pools).
• The image matches the Kubernetes version of the control plane.
• This mode is convenient but does not enable drift detection.
• Existing nodes are not automatically replaced after a control plane upgrade.

Option B - Pinned Image Mode

When imageOcid is specified on the OciNodeClass:

• Every node uses the specified image.
• Updating imageOcid triggers drift detection.
• Karpenter replaces outdated nodes while respecting disruption budgets.
• Ensure the pinned image is compatible with the control plane.

info

If maintaining a uniform Kubernetes version across all worker nodes is important, use a pinned image (imageOcid) instead of implicit image selection.

A practical approach is to copy the imageOcid from a validated, already-running node using the OCI console.

Update Kubernetes Add-ons

Update node-level add-ons that must remain version-aligned with the control plane, including:

• CNI
• kube-proxy
• CoreDNS
• Block Volume CSI Driver
• Other CSI drivers

Verify Pod Disruption Budgets

Ensure PodDisruptionBudgets (PDBs) are configured for critical workloads. Check the PDBs using the following command:

kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed --no-headers | awk '$3==0'

If a PDB returns 0, do one of the following:

• Scale the workload to ≥ 2 replicas
• Temporarily set maxUnavailable: 1 for the maintenance window, then revert.

Verify Capacity

Confirm sufficient quota and cluster capacity exists to run old and new nodes simultaneously during the upgrade.

(Optional) Enable Server-Side Instance Filtering

To allow server-side instance filtering, add the following IAM policy:

Allow dynamic-group <dynamic-group-name> to use search-resources in compartment <compartment-name>

Without this permission, Smart Karpenter falls back to listing and filtering all instances locally.

Step 1: Upgrade the Kubernetes Control Plane

warning

OKE control-plane upgrades are irreversible.

Upgrade the cluster's Kubernetes version (for example, using OKE).

Wait until the control plane reports a healthy status.

note

This step upgrades only the control plane. Existing worker nodes continue running their current kubelet version until they are replaced.

Step 2: Restart Smart Karpenter

Restart the Smart Karpenter deployment using the following commands:

kubectl rollout restart deployment <deployment/release name> -n <namespace>

kubectl rollout status deployment <deployment/release name> -n <namespace>

To get the deployment and namespace, use the following command:

kubectl get deploy -A | grep -i karpenter

Restarting Smart Karpenter ensures it:

• Refreshes cached cluster information
• Resolves the latest OKE image
• Launches new nodes using images compatible with the control plane

Even when using implicit image selection, you must still restart Smart Karpenter.

However, restarting alone does not replace existing nodes.

After restarting Smart Karpenter, verify the controller is healthy using the following command:

kubectl logs -n <namespace> -l app.kubernetes.io/instance=<release-name> --tail=100

Step 3: Recycle Worker Nodes

Worker nodes are not upgraded in place. The worker nodes must be replaced with newly provisioned nodes.

The following section describes the recommended methods to recycle worker nodes.

Method 1: Use rotateAfter

rotateAfter provides graceful and zero-downtime node rotation. This configuration property is set on the OciNodeClass and applies to all nodes from that class. To replace nodes one at a time during the rollover, also set the NodePool disruption budget to nodes: "1".

Example:

apiVersion: karpenter.multicloud.sh/v1alpha1
kind: OciNodeClass
metadata:
  name: default # replace default with your OciNodeClass name (kubectl get ocinodeclasses.karpenter.multicloud.sh)
spec:
  rotateAfter: "7d"

For testing:

rotateAfter: "10m"

Set expireAfter on the NodePool to a value greater than rotateAfter so graceful rotation occurs before forced expiration.

Method 2: Configure Disruption Budgets

Disruption budgets limit the number of nodes replaced simultaneously. Configure the disruption budgets as shown in the following example.

Example:

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  budgets:
    - nodes: "1"

Configure nodes in literal value and not in percentage.

consolidateAfter

consolidateAfter controls how long a node remains underutilized before the Smart Karpenter considers consolidation.

It does not delay eviction after replacement capacity is available.

Before reducing this value:

• Configure appropriate PodDisruptionBudgets.
• Configure the pod anti-affinity wherever necessary.
• Set terminationGracePeriodSeconds appropriately.
• Use NodeClaim.spec.terminationGracePeriod if a maximum drain duration is required.

Method 3: Temporarily Reduce expireAfter

If node replacement is not occurring quickly enough, temporarily reduce expireAfter on the NodePool. Combined with disruption budget nodes: "1", nodes are replaced one at a time. After the upgrade completes, restore the original value.

expireAfter is more disruptive than rotateAfter because it does not guarantee the new node → ready → drain → delete sequence.

Method 4: Manual Node Rotation

Drain nodes manually in small batches using the following commands:

kubectl cordon <node-name>

kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=15m 

If the drain stalls (which almost always happens when PDB has disruptionsAllowed=0), stop and

1. Uncordon the node to restore scheduling using the following command:

   kubectl uncordon <node-name>

2. Re-check PDBs (see Verify Pod Disruption Budgets) and resolve any PDB showing 0 before retrying.

   note

   Always pass --timeout. Without a timeout, kubectl drain hangs indefinitely on a single-replica PDB-protected pod.

Smart Karpenter provisions replacement nodes automatically when workloads become unschedulable.

Best Practices

• Avoid replacing every node simultaneously.
• Verify quotas before increasing rollout speed.
• Perform upgrades during low-traffic periods whenever possible.

Protect Critical Workloads

Prevent automatic eviction of critical pods by adding the following properties:

metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"

Nodes hosting these pods are not auto-recycled. Plan to drain and replace them manually on your own schedule.

Avoid Manual Drift Manipulation

Editing the following property may trigger drift but is not officially supported:

metadata.annotations.karpenter.sh/nodepool-hash

Instead, prefer:

• rotateAfter
• temporarily lowering expireAfter

Orphaned Instance Cleanup

Smart Karpenter v1.2.0 introduces an orphaned-instance garbage collection controller.

The controller:

• Removes OCI instances whose NodeClaims no longer exist
• Prevents unnecessary cloud costs
• Deletes one instance per reconciliation cycle

Cleanup may therefore take multiple cycles.

Step 4: Validate the Upgrade

During the Rollover

Validate the node replacement throughout the rollout using the following commands:

kubectl get nodes -o wide          # check the KUBELET-VERSION column
kubectl get nodeclaims             # see in-flight Karpenter provisioning
kubectl get pods -A | grep -vE 'Running|Completed'
kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed --no-headers

Post Node Replacement

After the node replacement completes:

• Verify every node reports the upgraded kubelet version.
• Confirm no nodes remain on the previous Kubernetes version.
• Run workload health checks.
• Review NodePool and OciNodeClass status conditions.
• Restore any temporary expireAfter values.
• Verify orphaned OCI instances have been cleaned up.