← Writing

Dependency order is the real EKS deployment problem

EKS · Kubernetes · Terraform · Platform

Every EKS rollout I’ve watched go badly went badly in the same way: the pieces were all correct, and they arrived in the wrong order.

This is not a Terraform problem or a CloudFormation problem. It’s a problem about what has to exist before what — and most tooling will happily let you declare a dependency graph that is subtly wrong, then fail sixteen minutes into an apply.

Three layers, not one

It helps to stop thinking of “the cluster” as one thing. There are three layers with genuinely different lifecycles:

  1. Substrate — VPC, subnets, route tables, endpoints, the IAM roles the control plane assumes. Changes rarely. Owned by whoever owns the network.
  2. Cluster — the control plane, node groups, the OIDC provider, add-ons that AWS manages. Changes at upgrade cadence.
  3. In-cluster platform — controllers, ingress, cert management, observability agents, policy. Changes weekly.

The failure mode is treating all three as one apply. When you do, a routine change to layer three can plan a replacement of something in layer one.

The ordering that actually matters

If a resource needs a Kubernetes API that doesn’t exist yet, no amount of depends_on will save you.

The subtle ordering constraints are the ones that cross the boundary between AWS and the cluster:

  • The OIDC provider must exist before any IRSA role can be trusted, and the role must exist before the controller that assumes it starts, or you get a pod that crash-loops on AccessDenied and looks like a network problem.
  • Node groups must be ready before anything with a Deployment schedules, which sounds obvious until an add-on installs during node group creation and half its pods land Pending forever.
  • The ingress controller’s IAM policy has to be complete before the first Ingress object is created, because a partially-permitted controller will create a load balancer and then fail to tag it — leaving you an orphan you pay for.

How to encode it

Split state by layer, then make the dependency explicit and one-directional: layer three reads outputs from layer two, layer two from layer one, and nothing ever reads upward. It’s slower to bootstrap and dramatically faster to debug — when an apply fails you already know which of three blast radii you’re in.

The test for whether you got it right: can you destroy and rebuild layer three at 3pm on a Tuesday without anyone noticing? If not, the boundary is in the wrong place.