The first time I inherited a multi-team delivery estate, production was reachable four different ways. One team still had a laptop AWS key in a password manager. Another had a Jenkins job nobody would edit, because the last person who tried spent a Friday undoing it. A third had copied a GitHub Actions workflow, then forked it until a pull request on the original could not be reviewed. Staging and production shared one Actions secret store. When something went wrong, rollback was whoever remembered the last image tag in Slack.
Actions was not the problem. The envelope had no owner. Workflows had grown around tickets, not around who gets paged when a deploy hangs at 11pm. Once you treat the reusable workflow as the platform contract, callers stay thin, Environments own the OIDC subject, and teams stop inventing a private path to production.
Three things called standardisation
A YAML file copied into five application repos is not standardisation. The copies diverge on the first incident. An unversioned snippet in a wiki is not either. Nobody reviews a wiki for a production deploy. A versioned workflow_call in org/platform-workflows, with CODEOWNERS on the templates, is the only shape a reviewer can actually hold. The other two are how the laptop key came back wearing YAML.
A concrete estate
A mid-size platform I keep coming back to looks like this:
- five application teams shipping into the same staging and production accounts
- four of those teams already on GitHub Actions, each with a private copy of something that used to be shared
- one leftover batch job still on Jenkins
- an org repo,
org/platform-workflows, that did not exist when I arrived
The mistake is to treat that as a tooling-choice problem, or to bless a second CI product because one team likes GitLab. Four workload templates: web, static, worker, IaC. One CI product for all new work: GitHub Actions. Jenkins is frozen. It is not a second blessed path.
What usually goes wrong
Delivery drift looks like a collection of local decisions. It is usually the same handful of modes.
- Forked workflow, no CODEOWNERS. Someone copies
deploy.ymlinto the service repo. Six months later the original author cannot review a change, because the original no longer exists as a thing anyone opens. AWS_ACCESS_KEY_IDin repo secrets. Long-lived keys in Actions are a finding, not a convenience. Staging and production sharing one secret store is how a staging leak becomes a production incident. A PAT used to push images is the same class of finding.:latestand a rebuild on the prod job. Production checks outmainand builds again. What ran in staging is already gone. Rollback is a guess.- One Actions secret store for staging and prod. If the staging workflow can read the production credentials, the Environment names are labels, not controls.
- Required checks that exist only in a wiki. A wiki loses to a cutover window. The gate has to live on a reusable test workflow the pull request actually calls.
- A second CI product for one team. GitLab or Circle "just for this service" is how the envelope dies. New work is Actions. The Jenkins leftover does not get a friend.
uses: ...@main. The caller tracks a moving branch. A template change on Friday is a production change on Monday with no pin and no review in the app repo.- Pinning every caller of your template to a commit SHA by default. You stop receiving security and patch fixes on that major. SHA pins on
org/platform-workflowsare debt: use them when the currentv3line has a regression you cannot take, or when new commits on that major are features you have explicitly declined. Write the reason in a comment on the caller, and take the pin off once the major is safe again. Do not confuse this with image digests. The app artifact still promotes bysha256on GHCR or ECR. Do not confuse it with third-party actions either: those use a version tag so patches on that line land. A SHA onaws-actions/configure-aws-credentialsis the same optional debt, not the default. permissions: write-all, or relying on a permissive org default. New organisations defaultGITHUB_TOKENto contents and packages read, not write-all. The failure is an explicitwrite-all, or an org setting that still grants write by default. Set apermissions:block. A GHCR push job needspackages: writeon that job and on the caller. The callee cannot exceed the caller.secrets: inheritinto a reusable workflow that did not need them. Inherit dumps the caller's entire secret map into the called workflow. Name the secrets the template declared, or pass none.
Reference trees
The platform repo holds the templates. The application repo holds thin callers.
org/platform-workflows/
.github/
CODEOWNERS
workflows/
test.yml
web.yml
static.yml
worker.yml
iac.yml
org/payment-service/
.github/
workflows/
ci.yml
staging.yml
promote.yml
Polyrepo or monorepo, the caller stays thin. Path filters belong to the team. The envelope does not. CODEOWNERS on the template repo is /.github/workflows/ @org/platform, not decoration. Four workload templates stay web, static, worker, IaC. test.yml is the required-check envelope, not a fifth product.
The reusable workflow
The template is a workflow_call. Secrets are named, never secrets: inherit. Staging and production are different GitHub Environments, which means different OIDC subjects. On an Environment job the classic shape is repo:org/payment-service:environment:staging versus repo:org/payment-service:environment:production, and therefore different deploy roles. Repositories created after 15 July 2026 (not GitHub Enterprise Server) use an immutable prefix, repo:org@OWNER-ID/name@REPO-ID:…, with the same Environment suffix. Read the token from a real run, or gh api /repos/{org}/{repo}/actions/oidc/customization/sub. Do not guess. DEPLOY_ROLE_ARN is an Environment variable, not a repository variable. A repo-level vars value would give both Environments the same role. The JWT also carries job_workflow_ref for the called workflow. If you want only org/platform-workflows/.github/workflows/web.yml to assume the role, lock that claim. The IAM side lives in CI/CD roles with OIDC. Do not paste those roles into this repo's callers.
# org/platform-workflows/.github/workflows/web.yml (illustrative)
on:
workflow_call:
inputs:
image:
required: true
type: string
digest:
required: false
type: string
environment:
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: ${{ inputs.image }}-${{ inputs.environment }}
cancel-in-progress: false
jobs:
build:
if: ${{ inputs.digest == '' }}
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@v4
- id: push
run: ./build-and-push --image "${{ inputs.image }}"
deploy:
needs: build
if: ${{ !cancelled() && (needs.build.result == 'success' || needs.build.result == 'skipped') }}
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
permissions:
contents: read
id-token: write
steps:
- uses: aws-actions/configure-aws-credentials@v4.3.1
with:
role-to-assume: ${{ vars.DEPLOY_ROLE_ARN }}
aws-region: eu-central-1
- run: ./deploy --image "${{ inputs.image }}@${{ inputs.digest || needs.build.outputs.digest }}"
Own templates in org/platform-workflows default to a moving major tag, so patches and security fixes on that major land without a coordinated bump. Never @main. First-party actions/checkout may stay on @v4. Third-party actions use a version tag, same idea, so patches on that line land. A SHA there is debt with a reason, not the default.
# org/payment-service/.github/workflows/ci.yml (illustrative)
on:
pull_request:
permissions:
contents: read
jobs:
test:
uses: org/platform-workflows/.github/workflows/test.yml@v3
# org/platform-workflows/.github/workflows/test.yml (illustrative)
on:
workflow_call:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: ./test
# org/payment-service/.github/workflows/staging.yml (illustrative)
on:
push:
branches: [main]
permissions:
contents: read
id-token: write
packages: write
jobs:
build-and-stage:
uses: org/platform-workflows/.github/workflows/web.yml@v3
with:
environment: staging
image: ghcr.io/org/payment-service
The staging caller grants packages: write so the reusable build job can push to GHCR with GITHUB_TOKEN. The callee cannot request a scope the caller did not grant. Promote does not build, so it does not grant packages: write.
Pin a commit SHA on your template only when you must stay on one exact revision. Say why, and treat it as debt:
# Stay on this SHA: v3.4.0 on the v3 line ships a cache bug we cannot take.
# Remove when v3.4.1 is on the v3 tag.
uses: org/platform-workflows/.github/workflows/web.yml@9f86d081884c7d659a2feaa0c55ad015a3bf4f1b
Pull requests call test.yml. They do not set an Environment and they do not call web.yml. github.event_name inside a workflow_call is workflow_call, so you cannot gate deploy on "this is a PR" from inside web.yml. Staging is push to main. The promote job is workflow_dispatch on main. The production Environment has required reviewers and a deployment-branch policy that names only that branch. Fork pull requests never reach it.
Promotion and rollback
Omit digest on staging so the template builds, pushes a sha256 to GHCR or ECR, and deploys after merge to main. Pass it on production so the template skips the build. Production is a workflow_dispatch (or an Environment wait timer). Rollback is dispatch of the previous digest, not a rebuild of main. That rolls back the image. It does not roll back a database migration, a queue contract, or config that never lived in the artifact.
# org/payment-service/.github/workflows/promote.yml (illustrative)
on:
workflow_dispatch:
inputs:
digest:
description: sha256 already running in staging
required: true
type: string
permissions:
contents: read
id-token: write
jobs:
production:
uses: org/platform-workflows/.github/workflows/web.yml@v3
with:
environment: production
image: ghcr.io/org/payment-service
digest: ${{ inputs.digest }}
sequenceDiagram
participant Eng as Engineer
participant PR as Pull request
participant Test as test.yml
participant Call as web.yml
participant Stg as Staging Environment
participant IdP as OIDC
participant Prod as Production Environment
Eng->>PR: open pull request
PR->>Test: test.yml@v3
Note over Test: no Environment
Eng->>Stg: merge to main
Stg->>Call: web.yml@v3 build digest, push registry
Call->>IdP: environment staging
IdP-->>Stg: staging deploy role
Eng->>Prod: workflow_dispatch digest
Note over Prod: required reviewers
Prod->>IdP: environment production
IdP-->>Prod: production deploy role
Note over Prod: same sha256, no rebuild
Figure 3. Pull requests run tests. Staging is merge. Production is a digest plus reviewers. Neither deploy job rebuilds.
Jenkins freeze
The leftover batch job stays on Jenkins until it needs a behaviour change. Freeze means no new jobs, no new plugins, no second CI product. Security updates to Jenkins core and the plugins already installed still land until the job is gone. Unpatched Jenkins is a finding, not a freeze. Credentials on that job are rotated or left to die with the job. When that job next needs a behaviour change, it moves onto the worker template. New services are Actions only. A freeze you cannot describe in those terms is just a job nobody wants to touch, which is how it got here.
Who owns what
| Concern | Owner | Artifact |
|---|---|---|
| Template repo and CODEOWNERS | Platform | org/platform-workflows, required reviewers on template PRs |
| Environment rules | Platform, plus the service owner on production | Staging and production Environments, required reviewers, deployment branches |
| Deploy-role IAM policy | App team, inside the platform boundary | Scoped role per Environment; trust and sub in the RBAC note |
| Audit | Platform / security operations | Org audit-log streaming into the SIEM, not the Actions UI |
When those owners blur, the templates become another private copy, and the audit trail dies the next time someone says "just look in the workflow run."
The IaC caller
The IaC template is a thin caller: plan on the pull request, apply only on the Environment that owns that root, pin the module ref. State buckets, workspaces, and backend isolation are in Terraform module patterns for multi-environment cloud estates. This note does not re-teach them.
What working looks like
Standardisation is holding when a new service can adopt an existing template with a handful of inputs, pull requests run ci.yml, the last production deploy was a digest that already ran in staging, rollback is dispatch of the previous digest, and a new joiner can explain the path from the caller files without a hallway tour.
It is not holding when every service has a temporary workflow, production credentials outlive the pipeline that created them, callers still track @main, or the template repo has no CODEOWNERS.
What to check Monday
- Grep
uses:for@main. There should be none. - Confirm callers use
@v3unless a commented SHA pin exists, with a reason, treated as debt. - Confirm third-party
uses:(notactions/*) are version tags, not@main. A SHA pin is debt with a reason, same as on your own templates. - Confirm the production Environment has required reviewers and a deployment-branch policy. Confirm fork pull requests cannot select it. Confirm pull requests call
ci.yml, notweb.yml. - Confirm staging and production OIDC roles differ. Match the token's actual
sub: classicenvironment:staging/environment:production, or the immutablerepo:org@id/name@id:prefix on repos created after 15 July 2026. - Confirm no
AWS_ACCESS_KEY_IDin Actions secrets, no registry PAT, and nosecrets: inheriton reusable workflow calls. Confirm the staging caller grantspackages: writeif it pushes to GHCR. - Confirm Jenkins has no new jobs this quarter. Confirm Jenkins core and installed plugins have current security updates.
- Confirm the last production deploy was an image digest already in the registry, not a rebuild of
main.
None of this needs a new CI product. It needs treating the reusable workflow as something with an owner, not a file that happens to live in five repos.
What this note does not cover
SLSA attestations, self-hosted runners, merge queues, Helm or Kubernetes layout, a new Tools page, GitLab as a third product, Windows runners. Those are real topics. They are not this envelope, and this note does not pretend they are solved.
Related work
This note is the delivery side of Data and delivery engineering. Module and environment boundaries for the IaC caller are in Terraform module patterns for multi-environment cloud estates. Deploy-role trust and the sub claim live in CI/CD roles with OIDC.