On a team I worked with, the Playwright recorder made it cheap to capture every click, so every click got captured. The suite went red on a CSS class rename and stayed red, and production shipped over it, because nobody read red as a stop any more. The tool was fine. Nobody owned what red meant.

A browser suite is a gate when the deploy job lists it in needs:, and not otherwise. The envelope in CI/CD standardisation across application teams already has two gates before a digest reaches production: the image ledger, and the data-plane ledger from Zero-downtime database migrations with GitHub Actions. This note is the third. It asks whether a human path still works on the digest staging is serving, and it is deliberately the smallest of the three.

A concrete estate

Same mid-size platform as the two notes above:

  • org/payment-service ships a web image to staging on merge, then a workflow_dispatch of a digest to production through promote.yml
  • a staging host that serves whatever digest web.yml last deployed there, and reports that digest on /api/health, because the deploy passes it in as an environment variable
  • seeded accounts, one per subscription tier, on a staging database the pipeline can reach
  • org/platform-workflows with test.yml, web.yml, worker.yml, static.yml, iac.yml, migrate.yml

Pull requests stay on test.yml. Playwright end-to-end tests against a live Environment run on staging, after the staging deploy and before promote. Not in production, not on a fork, not as a second CI product.

What goes wrong

Recording everything. The recorder makes a fifty-step test as cheap as a five-step one, so the suite grows by whatever someone clicked last week. Every one of those steps is a place a class rename or a copy change can turn the run red without anything being broken. Runtime climbs past the pipeline budget, someone adds retries, and green becomes a matter of attrition rather than evidence.

Sleeps instead of state. waitForTimeout(2000) passes on the fast runner and fails on the slow one, and the fix that gets merged is waitForTimeout(5000). Playwright already waits for the locator to be actionable and for the assertion to be true; a hard sleep is a guess dressed as a wait. A suite with sleeps in it cannot tell you whether the product is slow or the test is impatient.

Shared data, lucky order. Parallel workers on one seeded database pass in the order they happened to run and fail in the order they run next Tuesday. Prefixing created rows and deleting them in afterEach fixes most of it, until the suite runs against a hosted staging where the test process has no database URL, the cleanup silently becomes a no-op, and a customer list fills with prefixed rows nobody meant to keep. A cleanup that cannot reach the database is not a cleanup. You find out from the list, not from the run.

Green against the wrong build. The smoke runs against staging, staging is still serving last week's digest because the deploy hasn't finished, the smoke passes, and the run certifies a build it never touched. Nothing in a green check says which digest it was green for. That has to be asserted, not assumed.

Two suites, not one

The suite that gates is small on purpose. Five journeys: sign in, create the core object, the write path the money depends on, the export a customer would notice within the hour, and the one admin action that mutates live data. It runs on every staging deploy, in the deploy's needs:, against the digest that deploy just shipped.

The suite that covers is larger and does not gate. One project per subscription tier, three stages each: a setup project that signs in through the UI once and caches storageState, a core project for read paths, a mutating project for writes. Each tier owns its own seeded organisation, so tiers run as parallel jobs in a matrix without colliding; tests inside a tier share that organisation, so each job runs one worker. It runs nightly on a schedule, and a red there is a ticket in the morning, not a blocked release.

Selectors are role-first. getByRole and getByLabel verify the accessible name a user would see; getByTestId is for the elements whose copy changes with language or plan, not the default. Playwright's own guidance orders them the same way, and the trade is honest: role locators catch a broken label, test ids survive a rewrite.

Traces are retain-on-failure, not on-first-retry, so the attempt that failed is the one you open. The job uploads test-results/ whenever the run wasn't cancelled, so a test that failed once and passed on retry still leaves its trace behind; a clean run uploads nothing. A red smoke on GitHub with no trace attached is a job name and a stack line, and that is how red loses its meaning.

// playwright.config.ts (illustrative)
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',
  fullyParallel: false,
  workers: 1,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI ? [['list'], ['html', { open: 'never' }]] : 'list',
  use: {
    baseURL: process.env.E2E_BASE_URL ?? 'http://localhost:5173',
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
  },
  projects: [
    {
      name: 'setup-team',
      testMatch: /auth\.setup\.ts/,
    },
    {
      name: 'smoke',
      testMatch: /smoke\.spec\.ts/,
      dependencies: ['setup-team'],
      use: {
        storageState: 'tests/e2e/.auth/team.json',
      },
    },
  ],
});

// tests/e2e/smoke.spec.ts (illustrative)
import { test, expect } from '@playwright/test';

test('create a customer and a draft invoice', async ({ page }) => {
  await page.goto('/customers');
  await page.getByRole('button', { name: 'New customer' }).click();
  await page.getByLabel('Name').fill('[E2E] Smoke customer');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByRole('heading', { name: '[E2E] Smoke customer' })).toBeVisible();

  await page.getByRole('link', { name: 'New invoice' }).click();
  await page.getByTestId('line-search').fill('Consulting');
  await page.keyboard.press('Enter');
  await page.getByRole('button', { name: 'Save draft' }).click();
  await expect(page.getByText('Draft saved')).toBeVisible();
});

One test id in the whole journey, on a search box with no stable label. Everything else is what a user would read.

The Playwright smoke job in GitHub Actions

e2e.yml is a GitHub Actions workflow_call template in org/platform-workflows, the same mechanism as migrate.yml. Named secrets only. It refuses production by hostname, and it refuses to run until staging is serving the digest it was handed.

# org/platform-workflows/.github/workflows/e2e.yml (illustrative)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      digest:
        required: true
        type: string
      spec:
        required: true
        type: string
    secrets:
      E2E_USER:
        required: true
      E2E_PASSWORD:
        required: true
      E2E_DATABASE_URL:
        required: true

permissions:
  contents: read

concurrency:
  group: e2e-${{ inputs.environment }}
  cancel-in-progress: false

jobs:
  smoke:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Refuse production, require the candidate
        env:
          BASE_URL: ${{ vars.E2E_BASE_URL }}
          DIGEST: ${{ inputs.digest }}
        run: |
          host=$(node -e "console.log(new URL(process.env.BASE_URL).hostname)")
          case "$host" in
            app.example.com|www.app.example.com) echo "::error::production is not a test target"; exit 1 ;;
          esac
          served=$(curl -sS --max-time 20 "$BASE_URL/api/health" | jq -r .digest)
          [ "$served" = "$DIGEST" ] || { echo "::error::staging serves $served, not $DIGEST"; exit 1; }
      - run: npx playwright test "$SPEC"
        env:
          SPEC: ${{ inputs.spec }}
          E2E_BASE_URL: ${{ vars.E2E_BASE_URL }}
          E2E_USER: ${{ secrets.E2E_USER }}
          E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}
          E2E_DATABASE_URL: ${{ secrets.E2E_DATABASE_URL }}
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-${{ inputs.environment }}-${{ github.run_id }}
          path: test-results/
          if-no-files-found: ignore
          retention-days: 7

E2E_DATABASE_URL is there so the cleanup can reach the staging database, through a role that can delete prefixed rows and nothing else. Without it the prefixed rows stay. E2E_BASE_URL is an Environment variable, not a secret; a hostname is not a credential. cancel-in-progress is false for the same reason it is on migrate.yml: a second run queues behind a running one instead of killing it mid-write.

The caller adds one job to staging.yml, after the deploy, consuming the digest that deploy produced:

# org/payment-service/.github/workflows/staging.yml (illustrative, added job)
jobs:
  smoke:
    needs: build-and-stage
    uses: org/platform-workflows/.github/workflows/e2e.yml@v3
    with:
      environment: staging
      digest: ${{ needs.build-and-stage.outputs.digest }}
      spec: tests/e2e/smoke.spec.ts
    secrets:
      E2E_USER: ${{ secrets.E2E_USER }}
      E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}
      E2E_DATABASE_URL: ${{ secrets.E2E_DATABASE_URL }}

Promote is a separate dispatch, so needs: cannot reach across to it. promote.yml gets a first job that reads the check run instead. A job from a called workflow shows up as caller job / called job, so the name to look for is smoke / smoke:

# org/payment-service/.github/workflows/promote.yml (illustrative, added job)
jobs:
  verify-staging:
    runs-on: ubuntu-latest
    permissions:
      checks: read
    steps:
      - name: Require a green smoke for this commit
        env:
          GH_TOKEN: ${{ github.token }}
          SHA: ${{ inputs.sha }}
        run: |
          conclusion=$(gh api "repos/$GITHUB_REPOSITORY/commits/$SHA/check-runs" \
            --jq '.check_runs[] | select(.name == "smoke / smoke") | .conclusion')
          [ "$conclusion" = "success" ] || { echo "::error::no green smoke for $SHA"; exit 1; }

  promote:
    needs: verify-staging

promote.yml takes the commit alongside the digest for this. If typing both is a burden, the image's org.opencontainers.image.revision label carries the same commit and can be read from the registry instead.

That is the whole gate: one needs: on staging, one check-run read on promote. A red smoke stops the staging run; a missing green stops the promote. The nightly ladder touches neither.

Playwright in production: synthetics, not gates

This suite never runs against production, and the job enforces it by hostname rather than by convention. That is not the same as saying Playwright never touches production. Synthetic monitoring does, read-only journeys on a synthetic account, alerting on failure, owned by whoever owns the pager. It is a different suite with a different owner and it is never a promote gate. The mistake is not running a browser in production; it is running the mutating gate there, load-testing customers, and creating data nobody can delete.

Who owns what

ConcernOwnerArtifact
Journey listApp teamFive journeys, not the backlog
Selectors and flake triageApp teamRole-first locators, test ids where copy varies, traces on fail
Staging dataApp teamSeeded tier accounts, prefixed rows, cleanup with a database URL
Gate on promotePlatform envelope, service owner on productionsmoke in staging.yml, verify-staging in promote.yml
Nightly ladderApp teamTier matrix on a schedule; a red is a ticket, not a block
Production syntheticsOps, separate suiteRead-only journeys, synthetic account, alerting, never a gate

When the gate and the ladder are the same suite, the gate takes long enough that people stop waiting for it. When the smoke checks staging without asserting which digest it is serving, it certifies whatever was there. Both are fixed by the same discipline: the gate is five journeys, and it knows what it is testing.

What to check Monday

  • Count the journeys in the deploy-blocking suite. If it is more than a handful, the rest belong in the ladder.
  • Diff staging.yml: the smoke job needs the deploy job, and nothing gates on the ladder.
  • Curl /api/health on staging and compare the served digest to the last staging run. If they differ, the last green certified nothing.
  • Grep for waitForTimeout. Each hit is a guess about timing that the next runner will disprove.
  • Open the trace from the last red. If there isn't one attached to the run, fix the upload before the next red.
  • Check that the hosted run has a database URL, then check the staging customer list for prefixed rows.
  • Read who owns the production synthetics. If the answer is "the same suite," it is a gate running in production.

A browser suite is a gate when the deploy job lists it. Everything above is what makes that line safe to write.

What this note does not cover

Visual regression, every-browser matrices, Playwright as a load test, mobile device farms, and the seeded-account model itself, which belongs to the RBAC note.

The image ledger is CI/CD standardisation across application teams. The data-plane gate is Zero-downtime database migrations with GitHub Actions. The one-person version of this shape runs on Heftli: a five-test smoke, a six-tier ladder, and a hostname check that refuses production. If your browser suite is red and the deploy went out anyway, get in touch.