Isolated and passing was never proof the code worked — only that the suite hadn't been allowed to find out yet.
The setup
TEAMBOX's PHP test suite was split across roughly two dozen modules — finance, framework, sync, staff, project management, file storage, and more — and every one of them carried a bespoke dual-runner setup: a "Live" test class and a separate non-live test class per feature, each inheriting from one of two custom base classes that existed purely to work around that split. It ran, and it mostly passed, but it was two parallel test formats doing the same job, and nobody had ever run a module's Live and non-Live suites in the same process to see what happened.
The plan was to migrate everything to Pest 3 — one test format, one bootstrap, no more artificial live/unit split. What actually happened split into three phases, and each one mattered more than the last.
Phase one: one test format, not two
Every module's paired Live and non-Live test classes were merged into single Pest-style files, batch by batch, retiring both custom base classes entirely. This was mechanical work at real scale — the line count actually went down once duplicate Live variants were folded together, and it accounts for the large majority of the diff across both phases.
But the mechanical merge is exactly what exposed what running tests in isolation had been hiding. Nearly every module gave up at least one previously-invisible bug the moment its tests stopped running in separate, isolated processes: a field-cache poisoning issue in a core data class that crossed module boundaries, a mock/real class collision in the holiday-calendar code, another one in the notification-message class, and a third affecting an application-level class that had been silently affecting dozens of test files without ever failing a single one. None of these were things you'd find reading the code — they only existed as an interaction between tests that had never been allowed to run next to each other before.
A rewrite this size can't live in one person's head across weeks of work, so every fix and decision went into a running delivery doc for continuity between sessions. And "done" meant something specific: each migrated module was re-run against the full suite before being marked complete, not just declared finished because its own tests passed in isolation — which, given what the merge kept finding, would have meant very little.
Phase two: hardening it into something you can actually trust
The rewrite alone didn't make the suite trustworthy. Parts of it were still partially skipped and non-blocking in CI — which is a polite way of saying it was quietly hiding its own bugs, the same way the old dual-runner split had been. Un-skipping eighteen previously-disabled framework tests surfaced real ones: a database reconnect that silently dropped the session timezone on every request, a transaction-savepoint depth leak specific to skipped tests, and tests that had only ever passed because of the order they happened to run in — all invisible until something forced them to run standalone instead of riding along in a sequence that happened to work.
The fix for the order-dependency problem was a transaction-wrapping PHPUnit extension that rolls every test back inside its own transaction, replacing thousands of lines of hand-written TRUNCATE/DELETE cleanup code scattered across the whole suite. That manual cleanup had been quietly wrong in places — which is precisely the kind of gap that produces a test suite where the order tests run in secretly matters.
With that in place, the suite moved from a non-blocking, partially-skipped check to a required CI gate. Not a suggestion anymore — it has to stay green, the same way any other check that can fail a build does.
Phase three: the fix from phase two had its own bug
Auditing the transaction-wrapping extension properly turned up something uncomfortable: a second custom extension built alongside it had been subscribing to the wrong test-lifecycle event, so it had never actually run — silently, since the day it was written. Going deeper surfaced more of the exact same class of bug phase one had found: fixtures using a database command that implicitly commits, quietly voiding the rollback wrapped around them, and a helper that opened a fresh database connection on every call, orphaning whatever transaction was supposed to contain it.
The most interesting find wasn't in a test at all — it was in how the suite's own connection-retry handling behaved once real isolation put it under pressure. Hardening that path closed a gap that had been quietly wearing a "flaky test" costume for who knows how long.
With isolation finally real, the last piece was making the suite fast enough to actually want to run: on the branch, CI moved to five isolated parallel workers, each with its own database and storage sandbox, cutting wall time from roughly thirteen minutes to about five. The branch itself was still a work in progress — more on that below.
Phase four: the rewrite outgrew the plan
What started as three phases across roughly two dozen modules kept growing. By early September the same branch touched 2,071 files (+129,063 / −88,332 lines), and the suite it produces runs 16,136 Pest tests across 29 suites with zero failures, errors, warnings, or deprecations. In early September it was still an open pull request — a change that size, touching every test file in the codebase, earns a slow review rather than a fast one, and mainline still runs the old suite while it waits.
Two of the three custom test extensions built for the earlier phases had their own hidden failure: both checked instanceof against a class name that didn't exist, which PHP evaluates as false rather than throwing an error. Neither extension had ever actually run — not once, for their entire lifetime. Fixing the typo immediately surfaced 29 files quietly leaking fixture data into whatever test ran next.
The CI topology took five attempts, and none of the failed ones were hidden from the history: four parallel workers (reverted the same day — too flaky), a retreat to fully sequential, a two-phase split, five workers once storage was isolated per worker, and finally a switch from hand-split workers to the test runner's own process-parallel mode. The real blocker across most of those attempts was never the worker split — it was a shared object-storage bucket that forced every storage-touching suite to serialise behind it, leaving one worker running ten suites for 419 seconds while four others finished in under three minutes and then sat idle.
Every dead end stayed in the commit history — the fix came from correctly blaming shared storage, not the worker split.
One projection was wrong in an instructive way: it assumed each worker's process-bootstrap cost was the ~17 seconds measured locally. In CI it's closer to 36. Across 29 invocations that's over 1,000 seconds of pure startup against roughly 760 seconds of actual testing — startup, not load balance, had quietly become the dominant cost. That's what finally pointed at replacing the hand-split workers rather than continuing to tune them.
Getting to genuine parallelism also meant fixing ten order-dependencies that had predated the branch by a long way — hidden for years behind a per-suite process boundary, and only surfaced once which files shared a worker started varying from run to run. Fixing them held one clear, counter-intuitive pattern: global fixes mostly made things worse, and per-test fixes held. Resetting one shared value globally took the parallel run from 4 failures to 47; only two genuinely process-wide values — active language and time zone — turned out to be safe to reset centrally.
One coverage decision was made and written down rather than applied quietly: a single reporting test accounted for roughly 46% of the entire suite's runtime, most of it spent deliberately — computing an unoptimised reference path to prove a much faster path matches it. Trimming redundant coverage cut that file's time by more than a third without touching the comparison that actually mattered, and the trade is documented next to the code rather than hidden. When a later, unrelated fix changed the test count, the same trade was revisited and re-affirmed rather than quietly left to rot.
And on the front end
A Playwright end-to-end suite joined CI in the same window, and its early flakes both turned out to be real bugs rather than test flakiness — a popup menu forwarding scroll events to a handler that assumed an element target, and an icon font's missing weight causing a one-frame layout shift that made an automated click land on the wrong element. Both were root-caused and fixed at the source, not papered over with a retry.
Phase five: merged, with a stricter definition of passing
The branch merged on September 21. The gate is now one parallel run: over sixteen thousand tests on five workers, each with its own database clone, temp directory and storage bucket, in about seventy seconds.
The bigger change is what counts as a pass. A run that prints a warning, an error or leftover fixture data fails, even when every test is green. Five extensions enforce it: a transaction per test, a state-cache reset, config-drift detection, log-noise detection and a database checksum. The log-noise check later turned out to stop counting after the logger re-initialised, so a noisy run could report clean. That fix is in review now: a guard also needs a test that proves it can fail.
Some production changes rode along, each with its own test: shell commands got a hard timeout, a timed-out backup no longer leaves an unreadable archive, database sessions keep their time zone across a reconnect, and a profiling extension stays out of the production image. Review raised 26 items: 17 fixed, 6 did not survive checking, 3 folded in on request.
What actually mattered
The rewrite itself — Pest 3, one format, retired base classes — was necessary, but it wasn't the point. The bugs it surfaced were the real payoff, and they only showed up because of one specific, unglamorous decision: stop trusting that isolated-and-passing meant working, and actually run everything together.
"The tests pass" is a claim about a specific suite configuration, not a fact about the code underneath it. That applies recursively: phase two's own supposed fix for isolation turned out to be half-silent too, and it took a third pass to actually catch it. Phases two and three — the un-skipping, the transactional isolation, catching the isolation fix that wasn't isolating anything, the CI gate, the parallel runners — are what made "the tests pass" mean roughly what it's supposed to mean again, instead of describing a suite that had quietly stopped checking large parts of itself.
The same discipline applies to the branch itself: something this size doesn't get to skip review just because its own numbers look good. It merges when it's ready, not when the numbers say it's ready.