Flaky tests are a signal, not noise.

A flaky test passes and fails on the same code. The first time it fails, someone investigates. The fifth time, someone clicks rerun. By the twentieth, the team has learned that a red build means nothing in particular, and a real regression can slip through on the third retry.

That loss of trust is the real cost, and it compounds. Engineers stop reading failures. Automatic retries are added to keep the pipeline green, which hides new flakiness as it appears. At Google's scale, about 1.5% of all test runs reported a flaky result, and almost 16% of tests showed some level of flakiness, according to its testing blog in 2016. You do not need that scale to feel it; a few hundred browser tests are enough.

A flaky test is also not always a test problem. Sometimes the nondeterminism is in the product: a race between two requests, a cache that serves stale data under load, a missing database constraint. Treating every flake as noise means occasionally ignoring the only evidence of a real bug.

The definition gives you the detection method: a test is flaky if it both passed and failed on the same commit. If your CI keeps results per test (most runners can emit JUnit XML, and it is worth loading into a table), one query finds them:

-- Tests that both passed and failed on the same commit in the last 14 days.
SELECT test_id,
       count(*)       AS flaky_commits,
       max(last_seen) AS last_seen
FROM (
  SELECT test_id, commit_sha, max(finished_at) AS last_seen
  FROM test_results
  WHERE finished_at > now() - interval '14 days'
  GROUP BY test_id, commit_sha
  HAVING bool_or(passed) AND bool_or(NOT passed)
) AS mixed
GROUP BY test_id
ORDER BY flaky_commits DESC;

Rank by the number of commits affected, not by raw failure count: a test that failed forty times on one broken branch is broken, not flaky. Rerunning failed tests automatically is a fine detection mechanism, as long as a pass on retry is recorded and reported as a flake and never silently counted as a pass.

Once a test is known to be flaky, it should stop blocking merges, because a gate that fails at random teaches people to bypass gates. It should not stop running. Quarantine means the test runs in a separate, non-blocking job, its results are still recorded, and it has an owner and a date by which it is fixed or deleted.

With Playwright, tags and annotations carry the quarantine metadata in the test itself, where reviewers see it:

test("exports the monthly report", {
  tag: "@quarantine",
  annotation: [
    { type: "owner", description: "billing-team" },
    { type: "expires", description: "2026-11-15" },
    { type: "issue", description: "https://tracker.example.com/QA-482" },
  ],
}, async ({ page }) => {
  // ...
});

npx playwright test --grep-invert @quarantine   # blocking job
npx playwright test --grep @quarantine          # non-blocking job; results still stored

A short script in CI that fails when any expires annotation is in the past keeps the list honest.

Most flakiness comes from a short list of causes. An empirical study of flaky tests in open-source projects found the top three root causes to be asynchronous waits, concurrency, and test order dependency. It also found that most flaky tests (78%) were flaky from the moment they were written, which makes review and new-test stress runs the cheapest place to catch them.

Wait for conditions, not time

The most common fix is the simplest. A fixed sleep is a guess about how long something takes, and CI machines are slower and noisier than laptops. Wait instead for the condition a user would see. Playwright's web-first assertions do this by default: they retry until the condition holds or the timeout expires.

// Flaky: assumes saving always finishes within two seconds.
await page.click("#save");
await page.waitForTimeout(2000);
expect(await page.textContent(".toast")).toBe("Saved");

// Deterministic: waits for what the user actually sees, up to the assertion timeout.
await page.getByRole("button", { name: "Save" }).click();
await expect(page.getByRole("status")).toHaveText("Saved");

Control the clock

Anything that depends on the current time, such as expiry banners, "due today" badges, or reports for last month, will eventually run on a day that breaks it. Control time explicitly. In unit tests, pass a clock into the code under test. In browser tests, Playwright's clock API fixes the time the page sees, and setting timezoneId in the test configuration stops the runner's zone from changing the result.

test("shows the renewal banner seven days before expiry", async ({ page }) => {
  // The fixture account's subscription renews on 2026-03-31.
  await page.clock.setFixedTime(new Date("2026-03-24T09:00:00Z"));
  await page.goto("/account");
  await expect(page.getByRole("alert")).toContainText("Renews in 7 days");
});

• A test that passes on retry is a flake, recorded and visible, never a silent pass.
• A known flaky test leaves the blocking gate within a day and keeps running in quarantine.
• Every quarantined test has an owner and an expiry date, and an expired quarantine fails the build.
• Each fix records its root-cause category, so recurring causes become engineering work rather than individual chores.
• The flake rate is reviewed with build time and failure rate as a measure of the suite's health.

The aim is not zero flaky tests forever. It is a suite where red means something, so that when it fails, people read it. That is the standard we hold test automation work to.