Kopf’s @on.create handler fires on every kubectl apply, not just creation

Built a Kubernetes operator with Kopf that provisions databases and Redis users for each tenant. The @kopf.on.create handler generates a password, creates the resource, and stores credentials in SSM.

Tested it. Worked great. Then someone re-applied the same Tenant CR and every running pod lost its database connection.

The cause

Kopf tracks whether it has “seen” a resource using annotations on the object. But @kopf.on.create isn’t tied to the Kubernetes create event — it fires when Kopf’s internal state says “I haven’t processed this object yet.”

If the operator pod restarts, or if Kopf’s state is reset, or in certain race conditions, @on.create fires again for existing resources. And kubectl apply on an existing resource can trigger it too, depending on how Kopf’s diffing works with your handler configuration.

The footgun

@kopf.on.create('tenants')
def on_tenant_create(spec, name, **kwargs):
    password = generate_password()          # new password every time
    create_or_update_user(name, password)   # overwrites existing user
    store_in_ssm(name, password)            # overwrites existing secret

Re-apply the CR and:

1. New password generated
2. Database/Redis user password changed
3. SSM parameter overwritten
4. Running pods still have the old password
5. Everything breaks

The handler’s docstring even said “if the user already exists, its password is rotated.” That’s not idempotency — that’s a landmine.

The fix

Check if the resources already exist before doing anything:

@kopf.on.create('tenants')
def on_tenant_create(spec, name, **kwargs):
    if user_exists(name) and ssm_params_exist(name):
        logger.info(f"Tenant {name} already provisioned, skipping")
        return

    password = generate_password()
    create_user(name, password)
    store_in_ssm(name, password)

If the user exists in the backing service and the credentials are in SSM, there’s nothing to do. Don’t generate a new password. Don’t call modify. Don’t touch SSM. Log and return.

The principle

Operator handlers must be idempotent. “Idempotent” doesn’t mean “runs without errors the second time.” It means “produces the same result whether it runs once or ten times.” Generating a new password on every invocation is the opposite of idempotent — it’s actively destructive on re-run.

The only side effect the operator needs is putting credentials where workloads can find them. If they’re already there, it’s a no-op.