A rollback tested against nothing, and six copies in 23 minutes

The brief was two things for every app: a way to ask a running deploy which commit it is, and a way to put the previous one back.

The first half is small. A /version route returns the commit SHA as plain text with no-store, so the CDN in front cannot answer with an old one. It touches no session, no auth and no database. The SHA goes into the image as the last build argument, so a new commit rebuilds one layer.

The second half shipped as a workflow around kamal rollback, and nine minutes later a line went into the deploy docs saying the rollback had been tested. It could not have worked. Each app binds a fixed port behind nginx, and a hook before boot removes the running container to free that port. So at any moment there is exactly one container, and never an old one to return to. The plan for the first version had even noted that Kamal keeps the last five. It keeps them only when nothing deletes them.

kamal rollback also exits 0 when it fails, which is how a test of it could look like a pass.

The rollback now redeploys the old image by SHA from the registry, skipping the push. That has limits, and they are written down: the old image must still exist, it boots with today’s config and secrets, migrations are not undone, and there are a few seconds of downtime. Kamal pulls before the hook runs, so a missing image leaves the live container alone. The docs line claiming a test came out.

Then it went to six more apps, all merged within 23 minutes. Each had its own wrinkle. One is also sold as a kit, and its release script refuses any file that names the build machine, so the new workflow went on its deny list. One needed an RPC url in the rollback environment or its ticker would stop. One answered /version from a controller instead of a bare route.

Every copy shares one gotcha in its spec. Ruby’s YAML parser reads the key on: as the boolean true, so each test reads the trigger as workflow[true].