fizz.today

Terraform can’t show you what ESO will deliver to your pod

I was reviewing a PR that changed an ESO ExternalSecret. The plan showed kubernetes_manifest.external_secret[0] will be updated in-place with a blob of YAML diff. Nothing in the plan told me what environment variables the pod would actually have after the apply.

The ExternalSecret has two sources of truth. The data entries are explicit — each one maps an env var name to an SSM path. Those are visible in the terraform code. But the dataFrom.find block discovers parameters at runtime by walking an SSM path prefix and matching names against a regex. Terraform has no idea what that will resolve to. The plan shows the find configuration, not the find results.

Querying SSM at plan time

A terraform external data source can run a shell command during the plan. I used it to call aws ssm get-parameters-by-path with the same path prefix and filter that ESO uses at runtime:

data "external" "apiserver_feature_flags" {
  program = ["bash", "-c", <<-EOF
    PARAMS=$(aws ssm get-parameters-by-path \
      --path "/ramparts/dev/tenants/momcorp/apiserver" \
      --recursive \
      --query 'Parameters[?contains(Name, `FEATURE_FLAG_`)].Name' \
      --output text 2>/dev/null | tr '\t' '\n' | while read -r p; do
        basename "$p"
      done | sort | paste -sd',' -)
    jq -n --arg params "$${PARAMS:-none}" '{"params": $params}'
  EOF
  ]
}

Then an output that combines all three sources of env vars into one manifest:

output "apiserver_env_manifest" {
  value = {
    plain_env     = sort(keys(merge({ TENANT_NAME = "", ENVIRONMENT = "", PORT = "" })))
    ssm_secrets   = sort([for k, v in local.ssm_params : trimprefix(k, "apiserver/") if startswith(k, "apiserver/")])
    feature_flags = data.external.apiserver_feature_flags.result.params != "none" ? sort(split(",", data.external.apiserver_feature_flags.result.params)) : []
  }
}

The plan becomes the complete picture

Changes to Outputs:
  + apiserver_env_manifest = {
      + feature_flags = [
          + "FEATURE_FLAG_ENABLE_ADMIN",
          + "FEATURE_FLAG_ENABLE_AGENTS",
          + "FEATURE_FLAG_ENABLE_AUTH",
        ]
      + plain_env     = [
          + "ENVIRONMENT",
          + "PORT",
          + "TENANT_NAME",
        ]
      + ssm_secrets   = [
          + "AUTHSERVICE_BASE_URL",
          + "DATABASE_URL",
          + "LOG_LEVEL",
          + "REDIS_CACHE_HOST",
          + "SECRET_KEY",
          + "TENANT_HASH",
        ]
    }

Every env var the pod will have, grouped by source. When someone adds a feature flag via SSM, the next plan shows it in the manifest even though no terraform code changed. When someone removes one, it disappears from the output.

The external data source adds one AWS API call per plan — negligible latency. It runs with the same credentials terraform already has. The output diffs like any other terraform output, so you see adds, changes, and deletes in the plan.

Terraform manages the ExternalSecret spec. ESO manages the runtime resolution. Without the manifest output, reviewing a PR means trusting that the YAML will resolve to something sensible — you can’t see the payload. With it, the plan shows every env var the pod will have before anyone hits apply.

#terraform #kubernetes #eso #platformengineering