fizz.today

Gunicorn 25 broke my read-only container

The apiserver pods were logging [ERROR] Control server error: [Errno 30] Read-only file system on every boot. The workers still started, requests still served, but gunicorn’s new control server — added in 25.x — was silently broken.

The control server creates a Unix socket in the current working directory. Our CMMC-hardened security context sets readOnlyRootFilesystem: true on every container. The working directory is /app, which is inside the read-only root. The socket creation fails, gunicorn logs the error, and moves on. No crash, no restart, just a dead feature nobody asked for.

The fix is one environment variable:

GUNICORN_CMD_ARGS="--control-socket /tmp/gunicorn.sock"

Point the socket at /tmp, which is an emptyDir volume mount that exists specifically so read-only containers have somewhere to write. The error disappears and the control server works if you ever want it.

I found this buried in crash logs from a different problem — weasyprint loading in every worker was eating all the memory and killing nodes. Once I fixed the memory issue (bigger nodes, fewer workers), the control server error was the only thing left, and it was on every single boot across every tenant. Gunicorn 25.x introduced the control server. Nobody told the Dockerfile.

#kubernetes #gunicorn #containers #platformengineering