On a team I worked with, the Playwright recorder made it cheap to capture every click, so every click got captured. The suite went red on a CSS class rename and stayed red, and production shipped over it, because nobody read red as a stop any more. The tool was fine. Nobody owned what red meant.
A browser suite is a gate when the deploy job lists it in needs:. The shared pipeline already has two gates before a digest reaches production: the digest that ran in staging, and the migration order from zero-downtime database migrations. This note is the third. It asks whether a human path still works on the digest staging is serving, and it is deliberately the smallest of the three. Visual regression, every-browser matrices, Playwright as a load test, and mobile device farms are separate topics.
The staging estate under test
From the smoke suite's side, the estate from the first two notes looks like this:
org/payment-service, deployed to staging on every merge and promoted to production by digest through aworkflow_dispatchofpromote.yml- a staging host that serves whatever digest
web.ymllast deployed there, and reports that digest on/api/health, because the deploy passes it in as an environment variable - seeded accounts, one per subscription tier, on a staging database the pipeline can reach
org/platform-workflowswithtest.yml,web.yml,worker.yml,static.yml,iac.yml,migrate.yml
Pull requests stay on test.yml. Playwright end-to-end tests against a live Environment run on staging, after the staging deploy and before promote. They never run in production, on a fork, or through a second CI product.
Four habits that make a smoke result meaningless
Recording everything. The recorder makes a fifty-step test as cheap as a five-step one, so the suite grows by whatever someone clicked last week. Every one of those steps is a place a class rename or a copy change can turn the run red without anything being broken. Runtime climbs past the pipeline budget, someone adds retries, and green stops being evidence.
Sleeps instead of state. waitForTimeout(2000) passes on the fast runner and fails on the slow one, and the fix that gets merged is waitForTimeout(5000). Playwright already waits for the locator to be actionable and for the assertion to be true; a hard sleep is a guess. A suite with sleeps in it cannot tell you whether the product is slow or the test is impatient.
Shared data, lucky order. Parallel workers on one seeded database pass in the order they happened to run and fail when that order changes. Prefixing created rows and deleting them in afterEach fixes most of it, until the suite runs against a hosted staging where the test process has no database URL, the cleanup silently becomes a no-op, and a customer list fills with prefixed rows nobody meant to keep. The run stays green, and you find out from the customer list.
Green against the wrong build. The smoke runs against staging, staging is still serving last week's digest because the deploy has not finished, the smoke passes, and the run certifies a build it never touched. Nothing in a green check says which digest it was green for. The job has to assert it.
A gating suite and a coverage suite
The suite that gates is small on purpose. Five journeys: sign in, create the core object, the write path the money depends on, the export a customer would notice within the hour, and the one admin action that mutates live data. It runs on every staging deploy, in the deploy's needs:, against the digest that deploy shipped.
The suite that covers is larger and does not gate. One project per subscription tier, three stages each: a setup project that signs in through the UI once and caches storageState, a core project for read paths, a mutating project for writes. Each tier owns its own seeded organisation, so tiers run as parallel jobs in a matrix without colliding; tests inside a tier share that organisation, so each job runs one worker. It runs nightly on a schedule, and a red there becomes a ticket in the morning.
Selectors are role-first. Preferred Playwright locators like getByRole and getByLabel verify the accessible name a user would see; getByTestId is the exception, for elements whose copy changes with language or plan. Role locators break when a label changes; test ids survive a rewrite but drift from what the user reads.
Traces are retain-on-failure, not on-first-retry, so the attempt that failed is the one you open. The job uploads test-results/ whenever the run was not cancelled, so a test that failed once and passed on retry still leaves its trace behind; a clean run uploads nothing. A red smoke with no trace attached gives you a job name, a stack line, and nothing to debug.
// playwright.config.ts (illustrative)
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests/e2e',
fullyParallel: false,
workers: 1,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 1 : 0,
reporter: process.env.CI ? [['list'], ['html', { open: 'never' }]] : 'list',
use: {
baseURL: process.env.E2E_BASE_URL ?? 'http://localhost:5173',
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
},
projects: [
{
name: 'setup-team',
testMatch: /auth\.setup\.ts/,
},
{
name: 'smoke',
testMatch: /smoke\.spec\.ts/,
dependencies: ['setup-team'],
use: {
storageState: 'tests/e2e/.auth/team.json',
},
},
],
});
// tests/e2e/smoke.spec.ts (illustrative)
import { test, expect } from '@playwright/test';
test('create a customer and a draft invoice', async ({ page }) => {
await page.goto('/customers');
await page.getByRole('button', { name: 'New customer' }).click();
await page.getByLabel('Name').fill('[E2E] Smoke customer');
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByRole('heading', { name: '[E2E] Smoke customer' })).toBeVisible();
await page.getByRole('link', { name: 'New invoice' }).click();
await page.getByTestId('line-search').fill('Consulting');
await page.keyboard.press('Enter');
await page.getByRole('button', { name: 'Save draft' }).click();
await expect(page.getByText('Draft saved')).toBeVisible();
});
One test id in the whole journey, on a search box with no stable label. Everything else is what a user would read.
The Playwright smoke job in GitHub Actions
e2e.yml is a GitHub Actions workflow_call template in org/platform-workflows following Playwright CI best practices, the same mechanism as migrate.yml. Named secrets only. It refuses production by hostname, and it refuses to run until staging is serving the digest it was handed.
# org/platform-workflows/.github/workflows/e2e.yml (illustrative)
on:
workflow_call:
inputs:
environment:
required: true
type: string
digest:
required: true
type: string
spec:
required: true
type: string
secrets:
E2E_USER:
required: true
E2E_PASSWORD:
required: true
E2E_DATABASE_URL:
required: true
permissions:
contents: read
concurrency:
group: e2e-${{ inputs.environment }}
cancel-in-progress: false
jobs:
smoke:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 24
- run: npm ci
- run: npx playwright install --with-deps chromium
- name: Refuse production, require the candidate
env:
BASE_URL: ${{ vars.E2E_BASE_URL }}
DIGEST: ${{ inputs.digest }}
run: |
host=$(node -e "console.log(new URL(process.env.BASE_URL).hostname)")
case "$host" in
app.example.com|www.app.example.com) echo "::error::production is not a test target"; exit 1 ;;
esac
served=$(curl -sS --max-time 20 "$BASE_URL/api/health" | jq -r .digest)
[ "$served" = "$DIGEST" ] || { echo "::error::staging serves $served, not $DIGEST"; exit 1; }
- run: npx playwright test "$SPEC"
env:
SPEC: ${{ inputs.spec }}
E2E_BASE_URL: ${{ vars.E2E_BASE_URL }}
E2E_USER: ${{ secrets.E2E_USER }}
E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}
E2E_DATABASE_URL: ${{ secrets.E2E_DATABASE_URL }}
- uses: actions/upload-artifact@v7
if: ${{ !cancelled() }}
with:
name: playwright-${{ inputs.environment }}-${{ github.run_id }}
path: test-results/
if-no-files-found: ignore
retention-days: 7
E2E_DATABASE_URL is there so the cleanup can reach the staging database, through a role that can delete prefixed rows and nothing else. Without it the prefixed rows stay. E2E_BASE_URL is stored as an Environment variable instead of a secret, because a hostname is not a credential. cancel-in-progress is false for the same reason it is on migrate.yml: a second run queues behind a running one instead of killing it mid-write.
The caller adds one job to staging.yml, after the deploy, consuming the digest that deploy produced:
# org/payment-service/.github/workflows/staging.yml (illustrative, added job)
jobs:
smoke:
needs: build-and-stage
uses: org/platform-workflows/.github/workflows/e2e.yml@v3
with:
environment: staging
digest: ${{ needs.build-and-stage.outputs.digest }}
spec: tests/e2e/smoke.spec.ts
secrets:
E2E_USER: ${{ secrets.E2E_USER }}
E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}
E2E_DATABASE_URL: ${{ secrets.E2E_DATABASE_URL }}
Promote is a separate dispatch, so needs: cannot reach across to it. promote.yml gets a first job that reads the check run instead. A job from a called workflow shows up as caller job / called job, so the name to look for is smoke / smoke:
# org/payment-service/.github/workflows/promote.yml (illustrative, added job)
jobs:
verify-staging:
runs-on: ubuntu-latest
permissions:
checks: read
steps:
- name: Require a green smoke for this commit
env:
GH_TOKEN: ${{ github.token }}
SHA: ${{ inputs.sha }}
run: |
conclusion=$(gh api "repos/$GITHUB_REPOSITORY/commits/$SHA/check-runs" \
--jq '.check_runs[] | select(.name == "smoke / smoke") | .conclusion')
[ "$conclusion" = "success" ] || { echo "::error::no green smoke for $SHA"; exit 1; }
promote:
needs: verify-staging
promote.yml takes the commit alongside the digest for this. If typing both is a burden, the image's org.opencontainers.image.revision label carries the same commit and can be read from the registry instead.
That is the whole gate: one needs: on staging, one GitHub check run query on promote. A red smoke stops the staging run; a missing green stops the promote. The nightly ladder touches neither.
Playwright in production: synthetics, not gates
This suite never runs against production, and the job enforces that by hostname. That is not the same as saying Playwright never touches production. Synthetic monitoring does, read-only journeys on a synthetic account, alerting on failure, owned by whoever owns the pager. It is a different suite with a different owner and it is never a promote gate. The risk is the mutating gate running there: it load-tests customers and creates data nobody can delete.
Who owns what
| Concern | Owner | Artifact |
|---|---|---|
| Journey list | App team | Five journeys, not the backlog |
| Selectors and flake triage | App team | Role-first locators, test ids where copy varies, traces on fail |
| Staging data | App team | Seeded tier accounts, prefixed rows, cleanup with a database URL |
| Gate on promote | Platform, service owner on production | smoke in staging.yml, verify-staging in promote.yml |
| Nightly ladder | App team | Tier matrix on a schedule; a red is a ticket, not a block |
| Production synthetics | Ops, separate suite | Read-only journeys, synthetic account, alerting, never a gate |
What to check Monday
- Count the journeys in the deploy-blocking suite. If it is more than a handful, the rest belong in the ladder.
- Diff
staging.yml: the smoke job needs the deploy job, and nothing gates on the ladder. - Curl
/api/healthon staging and compare the served digest to the last staging run. If they differ, the last green certified nothing. - Grep for
waitForTimeout. Each hit is a guess about timing that the next runner will disprove. - Open the trace from the last red. If there is not one attached to the run, fix the upload before the next red.
- Check that the hosted run has a database URL, then check the staging customer list for prefixed rows.
- Read who owns the production synthetics. If the answer is "the same suite," it is a gate running in production.
Related work
For the pipeline this gate belongs to, read Standardising CI/CD pipelines across teams; for the migration order it runs after, Zero-downtime database migrations with GitHub Actions. The one-person version of this shape runs on Heftli: a five-test smoke, a six-tier ladder, and a hostname check that refuses production. If your browser suite is red and the deploy went out anyway, get in touch.