Your deploy workflow just tried to create a tenant that doesn’t exist

The all-tenants deploy workflow failed. Terraform planned 72 new resources for a tenant called tenant-sentin. The real tenant was tenant-sentin-42b1fbfc0a — the hash suffix is part of the naming convention. tenant-sentin without the suffix was a ghost from a previous provisioning run, destroyed weeks ago. Its SSM parameters were never cleaned up.

I assumed the fix was deleting one stale SSM path. So I listed the full tree to confirm:

aws ssm get-parameters-by-path \
  --path "/ramparts/dev/tenants/" --recursive \
  --query 'Parameters[].Name' --output text \
  | tr '\t' '\n' | cut -d/ -f5 | sort -u

Twenty names came back. Three were real tenants. The other seventeen were ghosts — test tenants from early provisioning runs, tenants destroyed during debugging sessions, a tenant created with the wrong name and immediately abandoned. 239 stale parameters across 20 dead tenants, every one of them a phantom that would have blown up the next deploy.

I hadn’t realized how the workflow discovers tenants. It lists SSM paths under /ramparts/dev/tenants/. Every directory it finds becomes a Terraform workspace. No cross-reference against Kubernetes namespaces, no existence check, no validation. If a path exists in SSM, it’s a tenant. The provisioning workflow writes parameters on creation. The destruction workflow doesn’t delete them. Discovery trusts whatever it finds.

Nobody designed SSM as the tenant registry. It became one by accident — two workflows with an implicit contract that nothing enforces. One writes, one reads, and the gap between them filled up with 239 parameters that should have been deleted months ago.

aws ssm get-parameters-by-path \
  --path "/ramparts/dev/tenants/tenant-sentin" --recursive \
  --query 'Parameters[].Name' --output text \
  | tr '\t' '\n' \
  | xargs -n 10 aws ssm delete-parameters --names

After cleaning all 20 dead tenants, the workflow ran in 57 seconds. Three real tenants, zero phantoms. The SSM tree finally matched reality — until the next time someone destroys a tenant and forgets that SSM doesn’t know about it.