Isolated and passing was never proof the code worked — only that the suite hadn't been allowed to find out yet.
The setup
TEAMBOX's PHP test suite was split across roughly two dozen modules — finance, framework, sync, staff, project management, file storage, and more — and every one of them carried a bespoke dual-runner setup: a "Live" test class and a separate non-live test class per feature, each inheriting from one of two custom base classes that existed purely to work around that split. It ran, and it mostly passed, but it was two parallel test formats doing the same job, and nobody had ever run a module's Live and non-Live suites in the same process to see what happened.
The plan was to migrate everything to Pest 3 — one test format, one bootstrap, no more artificial live/unit split. What actually happened split cleanly into two phases, and the second one mattered more than the first.
Phase one: one test format, not two
Every module's paired Live and non-Live test classes were merged into single Pest-style files, batch by batch, retiring both custom base classes entirely. This was mechanical work at real scale — the line count actually went down once duplicate Live variants were folded together, and it accounts for the large majority of the diff across both phases.
But the mechanical merge is exactly what exposed what running tests in isolation had been hiding. Nearly every module gave up at least one previously-invisible bug the moment its tests stopped running in separate, isolated processes: a field-cache poisoning issue in a core data class that crossed module boundaries, a mock/real class collision in the holiday-calendar code, another one in the notification-message class, and a third affecting an application-level class that had been silently affecting dozens of test files without ever failing a single one. None of these were things you'd find reading the code — they only existed as an interaction between tests that had never been allowed to run next to each other before.
A rewrite this size can't live in one person's head across weeks of work, so every fix and decision went into a running delivery doc for continuity between sessions. And "done" meant something specific: each migrated module was re-run against the full suite before being marked complete, not just declared finished because its own tests passed in isolation — which, given what the merge kept finding, would have meant very little.
Phase two: hardening it into something you can actually trust
The rewrite alone didn't make the suite trustworthy. Parts of it were still partially skipped and non-blocking in CI — which is a polite way of saying it was quietly hiding its own bugs, the same way the old dual-runner split had been. Un-skipping eighteen previously-disabled framework tests surfaced real ones: a database reconnect that silently dropped the session timezone on every request, a transaction-savepoint depth leak specific to skipped tests, and tests that had only ever passed because of the order they happened to run in — all invisible until something forced them to run standalone instead of riding along in a sequence that happened to work.
The fix for the order-dependency problem was a transaction-wrapping PHPUnit extension that rolls every test back inside its own transaction, replacing thousands of lines of hand-written TRUNCATE/DELETE cleanup code scattered across the whole suite. That manual cleanup had been quietly wrong in places — which is precisely the kind of gap that produces a test suite where the order tests run in secretly matters.
With that in place, the suite moved from a non-blocking, partially-skipped check to a required CI gate. Not a suggestion anymore — it has to stay green, the same way any other check that can fail a build does.
What actually mattered
The rewrite itself — Pest 3, one format, retired base classes — was necessary, but it wasn't the point. The bugs it surfaced were the real payoff, and they only showed up because of one specific, unglamorous decision: stop trusting that isolated-and-passing meant working, and actually run everything together.
"The tests pass" is a claim about a specific suite configuration, not a fact about the code underneath it. Phase two — the un-skipping, the transactional isolation, the CI gate — is what made that claim mean roughly what it's supposed to mean again, instead of describing a suite that had quietly stopped checking large parts of itself.