Kubernetes ResourceQuota needs 2x headroom for rolling updates, not 1x

Deploys were failing silently. The new ReplicaSet would scale to zero available, the old ReplicaSet stayed at one, and nothing happened. No crash loops, no OOMKills. Just a rollout frozen at “1 old / 0 new” forever.

The cause

The RollingUpdate strategy with maxSurge: 1 (the default) creates one new pod before terminating the old one. During that overlap window, both pods exist simultaneously. Both count against the namespace’s ResourceQuota.

We had three services per tenant namespace. Each requested 256Mi-512Mi of memory. The ResourceQuota was set to 1536Mi for requests.memory – just enough for steady-state. When a deploy triggered a rolling update, the surge pod pushed total requests over the quota. Kubernetes rejected the pod with Forbidden: exceeded quota. The rollout stalled waiting for a pod that could never be created.

Kubernetes didn’t fail. It enforced exactly what we configured. We just configured it for the wrong world.

The math

Steady-state: three pods requesting a combined 1280Mi. Quota: 1536Mi. Available headroom: 256Mi. Largest single pod request: 512Mi. The surge pod needed 512Mi but only 256Mi was available. Blocked.

If your quota is sized exactly to steady state, you’ve guaranteed your next deploy can deadlock.

The fix

# Before
spec {
  hard = {
    "requests.memory" = "1536Mi"
    "limits.memory"   = "3Gi"
  }
}

# After
spec {
  hard = {
    "requests.memory" = "2Gi"
    "limits.memory"   = "6Gi"
  }
}

The rule: quota for requests must be at least steady_state_total + max_single_pod_request. That’s the minimum to guarantee one rolling update can proceed. In practice, 2x steady-state gives enough room for concurrent deploys across multiple services without overprovisioning.

The emergency fix

The live cluster was patched immediately with kubectl edit resourcequota to unblock the stuck deploy. Then the Terraform resource was updated to codify the change permanently. Next terraform apply reconciled the two – no drift, no surprises.

Why this is easy to miss

ResourceQuota violations don’t show up as pod failures. The pod never gets created, so there’s no crash event, no log output, nothing in kubectl get pods. You have to check kubectl describe replicaset or kubectl get events to see the FailedCreate event with the quota exceeded message. If you’re only watching kubectl rollout status, you see a deployment that never completes with no explanation.

Sizing quotas for steady state is wrong. Deployments are transient states. Your quotas have to survive the transition.