Checkout kept returning 500 after a digest rollback I had signed off as safe. The previous sha256 was serving; the payments table was not the previous payments table. A NOT NULL column with no default had shipped in the same window as the new image, and rolling the artifact back did not un-run it. Staging had passed because staging already had the column.

The pipeline in Standardising CI/CD pipelines across teams promotes one digest through two Environments, and rollback is the previous sha256. That undoes the image and nothing else: schema changes, queued messages, and settings stored outside the image stay as they were. This second note in the series covers the rest: zero-downtime, rollback-safe database migrations sequenced by GitHub Actions, and the queue and config changes that ride with them.

Out of scope here: Helm or Kubernetes operators, service mesh, multi-region active-active schema, Flyway or Liquibase as a product, and RDS Blue/Green runbooks. Application tables stay out of Terraform; platform roots and their state are in Terraform module patterns for multi-environment cloud estates.

Two ledgers: image digest and database schema

A release that touches payment-service writes two ledgers, whether anyone named them or not.

  • The image ledger. GHCR or ECR sha256. Staging ran it. Production dispatch takes the same digest. Rollback is the predecessor. That is the CI/CD note.
  • The data-plane ledger. Schema, queue and event shapes, Parameter Store or AppConfig values, GitHub Environment variables that are not the image. Digest rollback does not rewind this ledger.

RDS Blue/Green is a third thing: cut over the database instance. It is not expand/contract for the application schema, and it does not make a digest rollback safe.

Two-column diagram: rolling back an image digest assumed to undo a database migration on the left, versus an image ledger and a separate schema, queue, and config ledger on the right
Figure 1. Digest rollback is one ledger. Schema, queues, and out-of-image config are another.

One service, four kinds of change

Same estate as the CI/CD note:

  • org/payment-service ships a web image through staging, then workflow_dispatch of a digest to production
  • a Postgres schema the service owns, outside Terraform
  • an SQS queue other workers already consume
  • a handful of runtime flags in SSM that ops have changed without a rebuild

org/platform-workflows already has test.yml, web.yml, worker.yml, static.yml, and iac.yml.

Two shortcuts break this estate: hanging a migrate step off every replica start in web.yml, and letting the deploy role ALTER TABLE. Rolling deploys keep the old digest and the new digest in traffic together. Additive schema has to exist before the new image requires it. Contract happens after the old image is gone.

Release steps that break the data plane

Migrate on boot. Five new tasks start and five migrations race for the same lock under PostgreSQL explicit table locks. One holds it, four time out, and the deploy fails its health check. The incident channel calls it an image problem because the image changed, but the deployment failed because the database migration locked the table during startup.

NOT NULL in the same digest. Old replicas are still in traffic when the column lands. Without a default, their inserts fail on a column they have never heard of. With a default, the inserts succeed and the change is additive. That one clause decides which phase the migration belongs in, and the opening incident was the first case.

Producer first on the queue. The new image emits a field. Old consumers that validate strictly are still draining, and they dead-letter what they cannot parse. Rolling the producer back stops new poison messages. It does not clean out the ones already sitting in the dead-letter queue.

Config in SSM, rollback on GHCR. The previous digest comes back. The parameter someone changed an hour earlier does not. Behaviour stays new while every dashboard says the release was reverted. Nothing in this pipeline records that those two changes were related, or which parameters a digest depended on. That gap stays open. A partial mitigation: when a digest ships with a required SSM or AppConfig change, record it in the deployment log or the release PR description: the parameter name, the old and new value, and whether rolling back the digest also requires reverting the parameter. It is a manual habit, and nothing in the workflow enforces it.

Reference repository structure for database migrations

migrate.yml is one more workflow_call template in the same repo, called the same way as worker.yml.

org/platform-workflows/
  .github/
    CODEOWNERS
    workflows/
      test.yml
      web.yml
      migrate.yml
      worker.yml
      static.yml
      iac.yml

org/payment-service/
  .github/
    workflows/
      ci.yml
      staging.yml
      expand.yml
      promote.yml
      contract.yml

Pull requests still call only ci.yml, with no Environment and no migration.

Expand, then the image, then contract

Expand is additive: nullable column, new table, optional JSON field the old consumer ignores. Old image keeps working. Contract (rename, drop a column, NOT NULL without a default, remove a field) waits until the old digest is out of the target Environment. In between, old and new processes coexist. That window is why the migration cannot ship in the same container start as the new code. Drained is a state you can read. On ECS it means the service reports no running tasks on the old task definition, and every message the old consumers had in flight has been deleted or has come back onto the queue after its visibility timeout. Until both are true, contract waits.

Queues are the inverse of a naive producer ship: consumers accept the new optional field first. Producers emit it after those consumers are the only ones left. Standard SQS is at-least-once, so handlers are idempotent or they are wrong. FIFO queues deduplicate inside their window; that changes nothing about the order above.

Runtime config outside the image needs the same check as schema. If you will not revert the parameter when you revert the digest, do not bundle that parameter change with a digest you might roll back.

Feature flags gate incomplete behaviour. They are not a substitute for expand and contract on the data plane.

The migrate job and its role

migrate.yml is a one-shot job that runs inside an Environment, serialized with concurrency control to prevent parallel runs. It takes named secrets only, never secrets: inherit. MIGRATE_ROLE_ARN is an Environment variable for the same reason DEPLOY_ROLE_ARN is in the CI/CD note. A role ARN is an identifier, so it does not need to be a secret; the OIDC trust policy is what protects it.

The migrate role can ALTER TABLE. The deploy role behind web.yml cannot, because its database user has no DDL grant, and if you are on the Aurora Data API, because it has no rds-data:ExecuteStatement either. The role's trust policy and sub condition are in RBAC in production cloud estates; this note does not repeat the HCL. The script reads the database credential from Secrets Manager with the migrate role, so nothing in the workflow file holds a password. Version pinning follows the CI/CD note. The job has no packages: write because it never pushes an image.

# org/platform-workflows/.github/workflows/migrate.yml (illustrative)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      phase:
        required: true
        type: string

permissions:
  contents: read
  id-token: write

concurrency:
  group: migrate-${{ inputs.environment }}
  cancel-in-progress: false

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.MIGRATE_ROLE_ARN }}
          aws-region: eu-central-1
      - run: ./migrate --phase "$MIGRATE_PHASE"
        env:
          MIGRATE_PHASE: ${{ inputs.phase }}

phase is expand or contract. Unknown values exit non-zero. The script must be idempotent: re-running expand after a partial failure must not apply contract. cancel-in-progress is false on purpose. A second run queues behind a running migration instead of killing it. Postgres rolls back a killed transaction, but CREATE INDEX CONCURRENTLY cannot run inside one, and a cancelled build leaves an invalid index that the next expand has to notice and drop.

On staging, needs: enforces the order: expand runs while the previous digest is still serving, then the new digest builds and deploys onto the expanded schema. Production is three separate clicks under reviewers: expand, then the existing promote.yml dispatch, then contract once the old digest has drained.

# org/payment-service/.github/workflows/staging.yml (illustrative)
on:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write
  packages: write
jobs:
  expand:
    uses: org/platform-workflows/.github/workflows/migrate.yml@v3
    with:
      environment: staging
      phase: expand
  build-and-stage:
    needs: expand
    uses: org/platform-workflows/.github/workflows/web.yml@v3
    with:
      environment: staging
      image: ghcr.io/org/payment-service

# org/payment-service/.github/workflows/expand.yml (illustrative)
on:
  workflow_dispatch:
permissions:
  contents: read
  id-token: write
jobs:
  production-expand:
    uses: org/platform-workflows/.github/workflows/migrate.yml@v3
    with:
      environment: production
      phase: expand

Production expand is workflow_dispatch on main, same Environment rules as promote: required reviewers, deployment-branch policy, no fork path. Run it before promote.yml. contract.yml is the same caller with phase: contract, run after drain as its own click.

Promote stays the CI/CD caller: digest in, no build, no packages: write, no migrate step inside web.yml.

Five-step release and database migration sequence: staging expand, staging deploy, production manual expand gate, production promote with coexistence, and production contract after drain
Figure 2. Pull requests still do not migrate. Staging expands, then ships the digest. Production expands, then promotes, then contracts after drain.

Who owns what

ConcernOwnerArtifact
Image digest promote and rollbackPlatform, service owner on productionweb.yml, promote.yml, GHCR or ECR sha256
Expand and contract jobsApp team, inside the platform boundarymigrate.yml@v3, expand.yml, contract.yml
Migrate-role IAM and DB grantsApp teamMigrate role and its database user can run DDL for this service only; the deploy role's user cannot
Queue consumer compatibilityApp team that consumesOptional fields first, required fields after drain
Out-of-image configApp team plus whoever may change SSM without a shipA note on the change: revert with the digest or not
Drain windowWhoever runs promoteECS service on the new task definition only; old consumers' in-flight messages deleted or timed out back to the queue
Terraform platform rootsPlatformModules and backends in the Terraform note, not Flyway in backend.tf

Ownership is contested at exactly one word: drained. If nobody owns saying it, contract runs on a guess.

Evidence the release order works

The process is working when anyone on the team can name the last production digest and whether expand already ran, when a digest rollback is allowed only if contract has not run, and when the deploy role's database user cannot run DDL. The warning signs are migrate running on container start, production expand living in a wiki, and an SSM change in the same hour as a digest that was later rolled back with no record.

What to check Monday

  • Check that web.yml never invokes ./migrate, and that migration runs only through migrate.yml with concurrency per Environment.
  • Compare the production migrate role with the production deploy role. They differ, and the deploy role's database user has no DDL grant.
  • Trace the last staging run: expand before build-and-stage. Trace the last production release: expand.yml before promote.yml, contract.yml only after drain.
  • Date the last digest rollback against the last contract. If contract already ran, the previous image is not a rollback plan.
  • Read the last queue change. No producer required a field while old consumers were still in the Environment.
  • List the SSM and Environment variables that changed with that ship, each marked revert-with-digest or not.
  • Search for a second CI product, a Flyway job on Jenkins, or secrets: inherit on migrate.yml. None should exist.

This note extends the pipeline in Standardising CI/CD pipelines across teams. The migrate role's trust policy follows the pattern in RBAC in production cloud estates, and platform state isolation is in Terraform module patterns for multi-environment cloud estates. This is the data-plane side of Data and delivery engineering. If your release has two ledgers and one owner, get in touch.