That Test isn’t Flaky. It’s Broken.

I’ve seen two different teams delete and redo entire test suites after they became too inconsistent to be useful. Both times the root cause was the same: tests were "flaky," so people reran them when they failed. As tests were added, both suites accumulated more and more debt until starting fresh seemed like the best option.

"Flaky" is a dangerous word for tests because it suggests solutions: rerun the failed test, build a system to automatically retry failed tests, and accept that the suite will occasionally fail. Re-running "flaky" tests might lead to a passing test suite today, but the long-term impact of that policy is a test suite that will get more and more unreliable over time. Without feedback that lets engineers know that they introduced a new problem, the failure rate will only ever increase. Even if nothing changes with the base error rate, simply adding more tests will make failures more likely. And as your engineering team grows, the test suite will be run more often, making it more likely that people will hit problems.

A test suite that doesn’t have intermittently failing tests is easy to keep that way. If an intermittently failing test gets added, it will be relatively simple to debug and fix because you’ll discover it faster.

Why? A 1/100,000 chance to fail doesn’t stay that way

Imagine 5% of your tests are written in a way that will fail 1/100,000 times. If you have 1,000 tests, what are the chances that your test suite as a whole will run successfully? Not too shabby: 99.95%. But as you add more tests, your chances of a successful test run start to go down. At 10,000 tests, your test suite has a 1/200 chance of failing: 1 - (1 − 1/10⁵)^{(.05 * 10,000)} = 0.5%.

1 in 200 test suite runs failing doesn’t sound terrible, but if you think about a team of 10, 100, or 1,000 engineers (and their agents) running tests over the course of a week, many engineers will encounter at least one failed test run over the course of the week. If you assume an engineer runs the full test suite 10 times per day, that means that over the course of the week, they only have a .995⁵⁰ = 77.8% chance to have every run succeed. Having each engineer have over a 1/5 chance to hit an intermittently failing test suite run over the course of a week is culturally insidious.

Even so, if only 1 in 200 test suite runs failed this way, it’d merely be annoying. But when you have a test suite that is known to be "flaky," it contributes to an engineering culture where people will re-run "flakes." That means that when bad tests are added that fail more often than 1/100,000 times, engineers are more likely to believe that they’re just getting unlucky and hitting errors rather than realize that the suite is now failing on 1/100 or 1/50 full runs.

How do you discover and fix intermittently failing tests?

My favorite tool to flush out intermittently failing tests is running the test suite on an hourly cron. The crux is using a commit that has already passed CI on master/main so that you can be sure that every failure represents a true problem of some sort. Once you have a way of discovering and tracking intermittent failures, it’s relatively straightforward to start driving the failure rate down.¹

An hourly cron isn’t enough to surface every 1/1,000,000 or even 1/100,000 test failure, but the failures that it does surface will often represent a class of problem: perhaps the fixture-resetting logic is flawed, perhaps there are a lot of latent time bugs in our system, or perhaps there’s a bad pattern for waiting for an element on the screen. These test failures surfaced by the cron runner are gifts that let you improve the codebase as a whole to make that particular problem less likely. Even when it’s not a codebase-wide problem, a test file that has one questionable pattern in it is likely to have more, so when a single test fails, it often makes sense to scan through the whole file for other tests that could fail in the future and chat with the team about any patterns that could be better.

The goal of the hourly cron runner is to flush out intermittent failures, so you want to make those failures more likely if you can. Run your tests backwards, intentionally try to run these tests on machines with more resource contention, inject random delays into database response times to try to surface race conditions, and generally try to make the classes of problems you want to drive down more likely so that you have more examples of things to fix.

The hourly cron will run into many of your intermittent failures, but engineers will run into a lot more of them over the course of their work. The right process to address these failures will vary by team size and ownership, but it’s crucial to have a well-understood process for handling a failure. If the person running into the broken test isn’t the right person to fix it because they don’t have the time or context, it’s best to at least comment out the test so that it can’t continue to fail and either pass it off to the right team or come back to it later. Unreliable tests can’t be allowed to remain in the codebase.

Zero is the best limit

The idea of retrying tests while tracking the failures is an appealing one. In theory, you’ll be able to fix the tests that fail, and CI will be more reliable. With the right tooling and engineering culture, I could see this working well, but building that culture, managing the failure rate, and creating the right tooling takes effort. At a certain scale, this sort of retry-and-track system might be inevitable, but my instinct is to put that off as long as I can.

As soon as you move away from a zero-tolerance policy for intermittent failures, test failures become a metric that you need to manage. The alerting scheme will need to deal with the spikiness of real life, attribution and ownership will be fuzzier, and you’ll have discussions about whether the alerting limit should go up. Teams will be pressured to defer fixes because the suite still passes, and there won’t be any feeling of urgency. All of this seems much harder to do well than having a bright-line rule that tests shouldn’t fail. As a stopgap while you’re trying to get an unreliable test suite under control, I could see retry-and-track being a useful tool to drive test reliability improvements, but I don’t think that it should be the desired end state for most teams.

A Field Guide to Flaky Broken Tests

Engineering culture isn’t just knowing why to do things. Engineering culture is what’s easy to do every day, and that depends on technical knowledge. When an engineer runs into a "flaky" test, the best intentions in the world won’t matter if they don’t know how to debug what’s wrong. Let’s chat through the things that contribute to an unreliable test suite.

Bad Test Assertions

If a test fails without giving engineers the information they need to figure out why it failed, it’s natural to want to rerun it.

Assertions should log out enough info to make failures simple to debug. In Jest, when a test assertion like expect(things.length).toBe(4) fails, the error message will be "Expected: 4 / Received: 5." That’s hard to debug! There’s no clue what the extra item actually is or whether the existing items are even correct.

Contrast that with the error message you get from a test assertion like expect(things).toHaveLength(4). When that assertion fails, it will log out the array of things so that you can clearly see what happened. This makes debugging easier and therefore more likely to happen.

● bad error message

    expect(received).toBe(expected) // Object.is equality

    Expected: 4
    Received: 5

    4 |
    5 | it("bad error message", () => {
    > 6 |   expect(things.length).toBe(4);
        |                         ^
    7 | });
    8 |
    at Object.toBe (test/example.test.js:6:25)

● good error message

    expect(received).toHaveLength(expected)

    Expected length: 4
    Received length: 5
    Received array:  [1, 2, 3, 4, 5]

    8 |
    9 | it("good error message", () => {
    > 10 |   expect(things).toHaveLength(4)
        |                  ^
    11 | });
    12 |
    13 |

    at Object.toHaveLength (test/example.test.js:10:18)

There’s a similar problem with checking a single field on an object. The debugging information from expect(thing.single_field).toBe(true) is far inferior to what you get from expect(thing).toEqual(expect.objectContaining({ single_field: true })). Being able to see at a glance that thing wasn’t the right object makes tracking down the cause of a flake or bug more straightforward. (That assertion pattern in Jest is a little long, so I’d recommend adding a custom matcher if that’s the assertion library you’re using.)

Bad test assertions don’t directly cause test failures; they just make the failures less likely to get fixed. The easier you can make fixing failed tests – better docs, training, error messages, and test tooling – the better your test suite will become.

Test Bleed

If you’re running tests in parallel or don’t have good resetting code, it’s easy for one test to "bleed" and cause another to fail. If your test assertions are sound, the reason for the failure will normally be obvious: you’ll see the extra or missing row in your error assertion message, and you can trace that to the flow that isn’t hooked into the global reset hooks. Fixing is generally a matter of updating code so that each test runner uses its runner_id to only talk to its own resources.

A few quick notes:

• Because of the heavy modern use of git worktrees, I’d recommend prefixing your container and database names with the directory name.
• Almost every test-specific afterEach hook that resets state is better as a global afterEach hook. If you have code that needs to be reset properly after every test, someone will inevitably forget to reset it, and test code is more ergonomic if resets are automatic.
• 18,957 tests in under 6 minutes has some notes on setting up integration tests that hit distinct databases for every test runner.

Time is Terrible

I’ve seen tests fail on the weekend, between 6pm and 7pm MDT (midnight UTC), on the 31st of the month, after daylight saving time, or when a function call starts a few milliseconds before the next minute and time ticks forward. Sometimes these failures will even be issues affecting production code. Writing good time-related code is hard! And if we culturally accept flaky tests, we might never discover that the tests were flagging a real issue.

The pattern I use to avoid some of these problems and proactively surface others is to lock time for the test suite to a random date sometime in the next year. Freezing the time lets us avoid problems with time ticking forward a second and causing issues. The random future date is an attempt to surface possible test failures related to time – not just validating the code for the date that the test was written on.

// When a test flakes, providing TEST_TIME_OVERRIDE allows us to see if the problem was date-related.
const timeOverride = process.env.TEST_TIME_OVERRIDE ? new Date(process.env.TEST_TIME_OVERRIDE) : getRandomTimeSometimeInTheNextYear();

console.log(`TEST_TIME_OVERRIDE for test run: ${timeOverride}`);

beforeEach(() => {
    timekeeper.freeze(timeOverride);
});

Race Conditions and Arbitrary sleeps

If you write your test code in a way that allows race conditions, your test suite will hit those race conditions, particularly in environments under heavy load like CI. The most telling sign that race conditions are possible in a codebase is the presence of code to wait for a process to complete:

kickOffAsyncThing();
await sleep(1000); // I think this should be long enough!
expect(things).toBe(good);

This leads to two terrible options:

1. Accepting a test that fails intermittently
2. Adding a wait that’s long enough to always succeed that slows down your test suite.

I’ve seen test code set up to wait for 5 seconds before checking a criterion. That’s an eternity. A slow test suite is a bad test suite. People will use it less, write fewer tests in it, and be less likely to improve it.

Most of the time, you should be able to refactor code so that the test can wait for the signal it actually cares about. This will often make your production code better too – when you call a function, it’s helpful to know when the function’s work is done.

On the backend, code that enqueues work to be done later is a common source of arbitrary sleeps. For most test code, I like the pattern of having the queue-worker’s task run synchronously after being enqueued. You’ll need this work to be complete before the next test runs anyway to make sure fixtures are clean, and it makes testing more ergonomic.²

Another common source of problems is waiting for an element to render on the screen. The normal fix is to check for the presence of the element in a loop with a timeout. This is a spot where thinking carefully about failure messages is essential; just knowing that an element wasn’t present isn’t enough information to make debugging easy!

CI-Specific Failures

In the same way that engineers may inadvertently make an already-unreliable test suite worse by adding more bad tests, changes to CI infra – like reducing the resources provisioned for CI instances or changing images – can make intermittent test suite failures more likely. I’ve seen engineering teams where these sorts of infra-caused problems are dismissed as "CI issues" rather than being treated the same as any other test failure. Culturally, the team as a whole is responsible for building a reliable test suite, which means a few crucial things:

1. Any potentially relevant changes to CI setup should be shared broadly so that people can be on the lookout for stability issues
2. Whoever owns CI should run training on how to debug CI-specific problems
3. CI should resemble local runs as much as possible so that it’s simple to reproduce problems locally
4. "Infra flakes" should be treated the same way as any other bug.

If a noisy neighbor can cause an OOM, it will happen again, and you never want to normalize rerunning failures. The fixes for these issues are different, but the cultural muscle should be the same.

The right fit for an agentic tool?

Because many intermittent test failures have easy-to-review rote fixes, an agentic tool could theoretically fix them on its own. At a previous company, I set up a coding agent to open PRs to fix broken tests that the cron test runner identified. It didn’t work as well as you might expect.

I wasn’t impressed with the ability of coding agents (at the time, Opus 4.5) to reliably and correctly reach for overarching solutions to intermittent test failures. Rather than realizing that there was a bug in the queue fixture resetting code, the agent would instead be likely to fix the single test that failed by adding extra guards and manual resets. Or by adding manual sleeps to slow down the test block and make race conditions less likely. And it would never realize that the right solutation to a problematic test file was chatting with the team that authored it about better options to structure their code to make it more testable.

And because it was often impossible for agents to reproduce failures, many of the PRs ended up being confidently confabulated nonsense that didn’t fix the problem with the tests or code. The coding agent would come up with a theory, lack an easy way to invalidate that theory, and ship a PR based on it. The most successful fixes from this system were for problems with time or ordering because there’d often be a simple way for the agent to reproduce the failure, so it could tell when the problem was solved.

I don’t want to give the impression that the agentic fixes were useless though! They were often able to one-shot simple problems, and creating a good set of instructions for agents meant that there was a good /debug-flaky-test skill available locally. I’m sure I’ll end up building similar systems in the future; it just wasn’t the panacea I was hoping for when I set it up.

A few key takeaways:

1. Bad test assertions made test failures harder to debug, which made the agent more likely to confabulate a fix. It’s helpful to prioritize instructions to improve assertions and logging when the right fix wasn’t obvious.
2. Because many of the fixes weren’t correct, it’s helpful to build an easy way to copy-paste a prompt to start debugging locally with appropriate context.
3. Without cultural buy-in for a system like this, you’ll build up a lot of soon-to-be-stale PRs. It’s important to have the agent tag an appropriate owner for any PR it generates so that the PRs don’t rot.

Teach them to yearn for the vast and dependable test suite

Until you’ve seen a once-dependable test suite slowly rot because "flaky" tests were rerun rather than treated as broken and fixed, my insistence that a one-in-a-million test failure rate is too high to tolerate might seem extreme. It is! But that mindset helps build a culture where people look for and celebrate fixes to other failures that are far more likely.

It’s the most natural thing in the world to hit a random failure when working on something, have it succeed when retried, and continue on. I’ve done it. You’ve done it. But the long-term impact of not fixing these intermittent failures is a codebase that gets harder and harder to work on over time. That’s not the codebase I want to build.

I want a codebase that people are excited to work in. One with fast feedback and clear error messages when something is broken. Fixing broken tests rather than rerunning them is a crucial part of realizing that vision.