fizz.today

ignore_changes doesn’t protect what isn’t in state yet

I merged a PR that added 39 SSM parameters across 3 services, each with lifecycle { ignore_changes = [value] }. I assumed Terraform would seed the defaults and never touch existing values. The other engineer couldn’t deploy for 3 days.

What I thought would happen

The SSM parameters already existed in AWS — set by hand with working database credentials, API keys, service URLs. My PR codified them in Terraform with placeholder values and overwrite = true:

resource "aws_ssm_parameter" "tenant_config" {
  for_each  = local.ssm_params
  name      = each.key
  type      = "SecureString"
  value     = each.value
  overwrite = true

  lifecycle {
    ignore_changes = [value]
  }
}

The mental model: Terraform creates the resource with whatever value is in the .tf file, but on subsequent applies it ignores value changes. Existing params are safe.

What actually happened

CI/CD ran terraform apply on merge. These 39 parameters weren’t in Terraform state yet. ignore_changes compares the current state to the desired state — but there was no current state. Every parameter was a create, not an update. Terraform wrote my placeholder values unconditionally, clobbering working config across three services.

The sequence:

  1. PR merges with ignore_changes = [value]
  2. CI runs terraform apply
  3. Terraform sees 39 new resources (not in state)
  4. ignore_changes has no state entry to compare against — it’s a no-op
  5. All 39 params get placeholder values
  6. Three services break simultaneously

SSM version history saved me

SSM Parameter Store keeps version history. Every parameter retained its previous value at version N-1. I walked through all 39, compared the Terraform-written value against the previous version, and restored the originals. Twenty minutes, nothing lost permanently.

# See what Terraform clobbered
aws ssm get-parameter-history \
  --name "/ramparts/dev/tenants/momcorp/apiserver/DATABASE_URL" \
  --with-decryption \
  --query 'Parameters[-2:].[Version,Value]'

Import before you merge

Import existing resources into state before the PR merges:

terraform import \
  'aws_ssm_parameter.tenant_config["/ramparts/dev/tenants/momcorp/apiserver/DATABASE_URL"]' \
  /ramparts/dev/tenants/momcorp/apiserver/DATABASE_URL

Once the resource is in state, ignore_changes has something to compare against. It works exactly as documented — on updates, not on creates.

No state, no ignore

ignore_changes is a state-level directive. No state, no ignore. If your CI/CD runs terraform apply on merge, the clobber happens before you get a chance to import. State imports are a pre-merge gate, not a post-merge cleanup.

#terraform #aws #ssm #debugging