Runbook

Kubernetes Cluster Upgrade Checklist

A Kubernetes upgrade checklist for EKS, GKE, AKS or self-managed clusters: removed APIs, add-ons, control plane, node pools, drains and post-upgrade checks.

  • Typical schedule per minor version upgrade
  • The checklist 5 sections, 22 items
  • Variables {{cluster}} {{current_version}} {{target_version}}
  • Start from CI
Use this template Free, no credit card. Imports the checklist into your Runbook organization.

Use this checklist for each Kubernetes minor version upgrade, such as moving a cluster from one minor release to the next. It works for managed services like Amazon EKS, Google GKE and Azure AKS and for self-managed clusters. Kubernetes releases new minor versions regularly and each one is supported for a limited time, so upgrades are routine work. They still break things when a removed API or an outdated add-on is missed.

Who owns it

A platform or DevOps engineer owns the run. Application teams own the items that check their workloads, so assign those items to them. Upgrade a non-production cluster first with the same checklist, and keep both runs as a record.

Common pitfalls

  • Removed APIs. Each release can remove deprecated API versions. Manifests, Helm charts and operators that still use them fail after the upgrade. Scan for them first with tools like Pluto or kubent, or check the API server's deprecated API metrics.
  • Skipping versions. The control plane can only move one minor version at a time. If you are several versions behind, plan several upgrades.
  • Forgotten add-ons. CNI, CoreDNS, kube-proxy, ingress controllers and CSI drivers have their own compatibility ranges. Upgrade them on purpose, not by accident.
  • Drains that hang. A PodDisruptionBudget that allows zero disruptions, or a single replica, blocks node drains. Check PDBs before you start.
  • Version skew. Nodes may lag the control plane only within the supported skew. Do not leave node pools behind for months.

The checklist: Upgrade {{cluster}} to Kubernetes {{target_version}}

Plan

  • Release notes and the deprecation guide for {{target_version}} are read and relevant changes listed
  • {{target_version}} is exactly one minor version above {{current_version}}
  • For managed clusters, the provider supports {{target_version}} and its upgrade notes are read optional

    EKS, GKE and AKS each document provider-specific changes, add-on versions and support end dates.

  • Maintenance window is agreed and application teams are informed
  • The same upgrade has been done on a non-production cluster first

Compatibility checks

  • Manifests, Helm charts and live objects are scanned for APIs removed in {{target_version}}

    Use a tool such as Pluto or kubent, and check the API server metric for requests to deprecated APIs.

  • Add-ons (CNI, CoreDNS, kube-proxy, CSI drivers) have versions compatible with {{target_version}}
  • Ingress controller, cert-manager, operators and other cluster tools support {{target_version}}
  • kubectl, Terraform or OpenTofu providers and CI tools work with {{target_version}}
  • PodDisruptionBudgets allow at least one pod to be evicted for each workload
  • Workloads that must stay available have more than one replica

Backup

  • Cluster state is backed up, such as an etcd snapshot or a tool like Velero

    Managed services back up the control plane for you, but not your persistent volumes or your own objects.

  • Persistent volumes for stateful workloads have recent snapshots optional

Upgrade

  • Control plane of {{cluster}} is upgraded to {{target_version}}
  • Add-ons are upgraded to their versions for {{target_version}}
  • Node pools are upgraded one at a time with cordon and drain, or replaced with new nodes
  • All nodes report {{target_version}} and are Ready

Verify

  • All system pods in kube-system and add-on namespaces are running
  • Application pods are running and no pods are stuck in Pending or CrashLoopBackOff
  • Ingress, DNS and certificates work for a sample of services
  • Monitoring, logging and autoscaling work after the upgrade
  • Cluster version, date and any issues are recorded in the platform docs
Checklist as Markdown

Start this checklist from your pipeline

When the upgrade pull request for the cluster config is opened, POST /api/v1/runs with template "Upgrade {{cluster}} to Kubernetes {{target_version}}".

curl -X POST https://runbook.metastic.ch/api/v1/runs \
  -H "Authorization: Bearer $RUNBOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"template": "Upgrade {{cluster}} to Kubernetes {{target_version}}", "variables": {"cluster": "…", "current_version": "…", "target_version": "…"}}'

See the API and the GitHub Action

Frequently asked questions

Can I skip Kubernetes versions when upgrading?

Not for the control plane. You upgrade it one minor version at a time, and check removed APIs and add-on compatibility at each step. Nodes can lag behind the control plane within the supported version skew, but you should bring them up to date soon after.

How do I find deprecated APIs before a Kubernetes upgrade?

Scan your manifests and Helm charts with a tool such as Pluto or kubent, and check what live objects in the cluster use. The API server also exposes a metric that counts requests to deprecated APIs. On EKS, GKE and AKS, the console can show upgrade insights that flag them too.

What is different about an EKS upgrade?

The control plane upgrade is managed by AWS, but you still upgrade the managed add-ons, such as the VPC CNI, CoreDNS and kube-proxy, and your node groups yourself. Check the EKS version notes for each release, as AWS documents provider-specific changes there. The rest of this checklist applies unchanged.

Updated

Send feedback

Ideas, bugs, questions: we read every message.