Checkout kept returning 500 after a digest rollback I had signed off as safe. The previous sha256 was serving; the payments table was not the previous payments table. A NOT NULL column with no default had shipped in the same window as the new image, and rolling the artifact back did not un-run it. Staging had passed because staging already had the column.

The envelope in CI/CD standardisation across application teams is doing its job. One digest, two Environments, rollback is the previous sha256. That rolls back the image. It does not roll back a database migration, a queue payload, or config that never lived in the artifact. This note is the rest of that sentence: zero-downtime, rollback-safe database migrations sequenced by GitHub Actions, and the queue and config changes that ride with them.

Two ledgers: image digest and database schema

A release that touches payment-service writes two ledgers, whether anyone named them or not.

  • The image ledger. GHCR or ECR sha256. Staging ran it. Production dispatch takes the same digest. Rollback is the predecessor. That is the CI/CD note.
  • The data-plane ledger. Schema, queue and event shapes, Parameter Store or AppConfig values, GitHub Environment variables that are not the image. Digest rollback does not rewind this ledger. Pretending it does is how you get a healthy deploy and a broken checkout.

RDS Blue/Green is a third thing: cut over the database instance. It is not expand/contract for the application schema, and it does not make a digest rollback safe.

Two-column diagram: rolling back an image digest assumed to undo a database migration on the left, versus an image ledger and a separate schema, queue, and config ledger on the right
Figure 1. Digest rollback is one ledger. Schema, queues, and out-of-image config are another. Mixing them is how staging "passed" and production checkout still 500'd.

A concrete estate

Same mid-size platform as the CI/CD note:

  • org/payment-service ships a web image through staging, then workflow_dispatch of a digest to production
  • a Postgres schema the service owns, not a Terraform module
  • an SQS queue other workers already consume
  • a handful of runtime flags in SSM that ops have changed without a rebuild
  • org/platform-workflows already has test.yml, web.yml, worker.yml, static.yml, iac.yml

The mistake is to hang a migrate step off every replica start in web.yml, or to let the deploy role ALTER TABLE. Rolling deploys keep the old digest and the new digest in traffic together. Additive schema has to exist before the new image requires it. Contract happens after the old image is gone.

What goes wrong

Migrate on boot. Five new tasks start and five migrations race for the same lock. One holds it, four time out, and the deploy fails its health check. The incident channel calls it an image problem because the image is what changed. The migration was the change; the image was the messenger.

NOT NULL in the same digest. Old replicas are still in traffic when the column lands. Without a default, their inserts fail on a column they have never heard of. With a default, the inserts succeed and the change is additive. That one clause decides which phase the migration belongs in, and the opening incident was the first case.

Producer first on the queue. The new image emits a field. Old consumers that validate strictly are still draining, and they dead-letter what they cannot parse. Rolling the producer back stops new poison messages. It does not clean out the ones already sitting in the dead-letter queue.

Config in SSM, rollback on GHCR. The previous digest comes back. The parameter someone changed an hour earlier does not. Behaviour stays new while every dashboard says the release was reverted. Nothing in this envelope records that those two changes were related, or which parameters a digest depended on. That gap stays open.

Reference trees

migrate.yml is a workload job, not a fifth CI product. Four templates stay web, static, worker, IaC. It is a workflow_call template, the same mechanism worker.yml already uses.

org/platform-workflows/
  .github/
    CODEOWNERS
    workflows/
      test.yml
      web.yml
      migrate.yml
      worker.yml
      static.yml
      iac.yml

org/payment-service/
  .github/
    workflows/
      ci.yml
      staging.yml
      expand.yml
      promote.yml
      contract.yml

Pull requests still call ci.yml only. They do not set an Environment. They do not migrate.

Expand, then the image, then contract

Expand is additive: nullable column, new table, optional JSON field the old consumer ignores. Old image keeps working. Contract (rename, drop a column, NOT NULL without a default, remove a field) waits until the old digest is out of the target Environment. In between, old and new processes coexist. That window is the reason the order is not "ship the migration in the same container start as the new code." Drained is a state you read, not a feeling. On ECS it means the service reports no running tasks on the old task definition, and every message the old consumers had in flight has been deleted or has come back onto the queue after its visibility timeout. Until both are true, contract waits.

Queues are the inverse of a naive producer ship: consumers accept the new optional field first. Producers emit it after those consumers are the only ones left. Standard SQS is at-least-once, so handlers are idempotent or they are wrong. FIFO queues deduplicate inside their window; that changes nothing about the order above.

Runtime config that is not in the image gets the same Monday question as schema. If you will not revert the parameter when you revert the digest, do not bundle that parameter change with a digest you might roll back.

Feature flags gate incomplete behaviour. They are not a substitute for expand and contract on the data plane.

The migrate workflow

migrate.yml is a one-shot job that runs inside an Environment. It takes named secrets only, never secrets: inherit. MIGRATE_ROLE_ARN is an Environment variable, not a repository variable, because a repository-level vars value would hand staging and production the same role. It is a variable, not a secret: a role ARN is an identifier, and the OIDC trust policy is what protects it. The migrate role can ALTER TABLE. The deploy role behind web.yml cannot, because its database user has no DDL grant, and if you are on the Aurora Data API, because it has no rds-data:ExecuteStatement either. The role's trust policy and sub condition are in RBAC patterns; this note does not repeat the HCL. The script reads the database credential from Secrets Manager with the migrate role, so nothing in the workflow file holds a password. Own templates are called at @v3. Third-party actions use a full version tag; pinning to a commit SHA is the exception, reserved for a known bad tag or an active incident. First-party actions/checkout stays on @v4. The job has no packages: write: it never pushes an image.

# org/platform-workflows/.github/workflows/migrate.yml (illustrative)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      phase:
        required: true
        type: string

permissions:
  contents: read
  id-token: write

concurrency:
  group: migrate-${{ inputs.environment }}
  cancel-in-progress: false

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v6.2.4
        with:
          role-to-assume: ${{ vars.MIGRATE_ROLE_ARN }}
          aws-region: eu-central-1
      - run: ./migrate --phase "$MIGRATE_PHASE"
        env:
          MIGRATE_PHASE: ${{ inputs.phase }}

phase is expand or contract. Unknown values exit non-zero. The script must be idempotent: re-running expand after a partial failure must not apply contract. cancel-in-progress is false on purpose. A second run queues behind a running migration instead of killing it. Postgres rolls back a killed transaction, but CREATE INDEX CONCURRENTLY cannot run inside one, and a cancelled build leaves an invalid index that the next expand has to notice and drop.

On staging the order is enforced by needs:, not by anyone remembering it: expand runs while the previous digest is still serving, then the new digest builds and deploys onto the expanded schema. Production is three separate clicks under reviewers: expand, then the existing promote.yml dispatch, then contract once the old digest has drained.

# org/payment-service/.github/workflows/staging.yml (illustrative)
on:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write
  packages: write
jobs:
  expand:
    uses: org/platform-workflows/.github/workflows/migrate.yml@v3
    with:
      environment: staging
      phase: expand
  build-and-stage:
    needs: expand
    uses: org/platform-workflows/.github/workflows/web.yml@v3
    with:
      environment: staging
      image: ghcr.io/org/payment-service

# org/payment-service/.github/workflows/expand.yml (illustrative)
on:
  workflow_dispatch:
permissions:
  contents: read
  id-token: write
jobs:
  production-expand:
    uses: org/platform-workflows/.github/workflows/migrate.yml@v3
    with:
      environment: production
      phase: expand

Production expand is workflow_dispatch on main, same Environment rules as promote: required reviewers, deployment-branch policy, no fork path. Run it before promote.yml. contract.yml is the same caller with phase: contract, after drain, not in the same click as promote.

Promote stays the CI/CD caller: digest in, no build, no packages: write, no migrate step inside web.yml.

sequenceDiagram
    participant Eng as Engineer
    participant Stg as Staging
    participant Mig as migrate.yml
    participant Web as web.yml
    participant Prod as Production

    Eng->>Stg: merge to main
    Stg->>Mig: phase expand
    Mig-->>Stg: expand complete
    Stg->>Web: build digest, deploy
    Note over Stg: new digest on expanded schema
    Eng->>Prod: workflow_dispatch expand
    Note over Prod: required reviewers
    Eng->>Prod: workflow_dispatch digest
    Note over Prod: old and new digests coexist
    Eng->>Prod: workflow_dispatch contract
    Note over Prod: after old digest drained

Figure 2. Pull requests still do not migrate. Staging expands, then ships the digest. Production expands, then promotes, then contracts after drain.

Who owns what

ConcernOwnerArtifact
Image digest promote and rollbackPlatform envelope, service owner on productionweb.yml, promote.yml, GHCR or ECR sha256
Expand and contract jobsApp team, inside the platform boundarymigrate.yml@v3, expand.yml, contract.yml
Migrate-role IAM and DB grantsApp teamMigrate role and its database user can run DDL for this service only; the deploy role's user cannot
Queue consumer compatibilityApp team that consumesOptional fields first, required fields after drain
Out-of-image configApp team plus whoever may change SSM without a shipA note on the change: revert with the digest or not
Drain windowWhoever runs promoteECS service on the new task definition only; old consumers' in-flight messages deleted or timed out back to the queue
Terraform platform rootsPlatformModules and backends in the Terraform note, not Flyway in backend.tf

When migrate lives in web.yml "to keep it simple," you have coupled a lock to every scale event. When contract ships in the same dispatch as the new digest, you have coupled a one-way schema step to an artifact you still claim you can roll back. Ownership is contested at exactly one word: drained. If nobody owns saying it, contract runs on a guess.

What working looks like

It is holding when Monday can name the last production digest and whether expand already ran, when a digest rollback is allowed only if contract has not run, and when the deploy role's database user cannot run DDL.

It is not holding when migrate runs on container start, when production expand is a wiki step, or when SSM changed in the same hour as a digest you later rolled back with no record.

What to check Monday

  • Check that web.yml never invokes ./migrate, and that migration runs only through migrate.yml with concurrency per Environment.
  • Compare the production migrate role with the production deploy role. They differ, and the deploy role's database user has no DDL grant.
  • Trace the last staging run: expand before build-and-stage. Trace the last production release: expand.yml before promote.yml, contract.yml only after drain.
  • Date the last digest rollback against the last contract. If contract already ran, the previous image is not a rollback plan.
  • Read the last queue change. No producer required a field while old consumers were still in the Environment.
  • List the SSM and Environment variables that changed with that ship, each marked revert-with-digest or not.
  • Search for a second CI product, a Flyway job on Jenkins, or secrets: inherit on migrate.yml. None should exist.

The tooling already exists. What is missing in most estates is an owner for the second ledger, the same way the reusable workflow already has one.

What this note does not cover

Helm or Kubernetes operators, service mesh, multi-region active-active schema, Flyway or Liquibase as the product, RDS Blue/Green runbooks, SLSA. Those are real topics. They are not this envelope.

Two things that look like they belong here and do not. Application tables in Terraform: the IaC caller applies platform roots, and state isolation for those roots is in Terraform module patterns for multi-environment cloud estates. A second CI product for migrations: it is still GitHub Actions, and the frozen Jenkins job from the CI/CD note does not grow a Flyway friend.

The image ledger is CI/CD standardisation across application teams. Module and state isolation for platform roots are in Terraform module patterns for multi-environment cloud estates. Migrate-role trust and the sub claim live in RBAC patterns. This note is the data-plane side of Data and delivery engineering. If your release has two ledgers and one owner, get in touch.