Core Trimmed Its PHPUnit Matrix. Do the Same to Yours.

WordPress core cut its PHPUnit test matrix by roughly 52 percent fewer jobs and 54 percent fewer job-minutes per run, ahead of 7.1. It kept full PHP version coverage while doing it. Separately, the proportion of runs needing a rerun to pass fell from about 68 percent to about 36 percent.

That second number is the one worth your attention. Halving CI cost is a budget win. Halving the rate at which your test suite lies to you is an engineering win, and it is the one most plugin suites need more.

The work landed across a handful of pull requests on WordPress/wordpress-develop: #12719 for trunk and #12720 for the 7.0 branch on the matrix trimming, #12701 for the Gutenberg build, #12703 for Docker pull retries, and #12726 backporting the matrix work to 6.8. None of it changes the test framework itself, so nothing here forces a change in your plugin. The reasoning is what transfers.

What core actually changed

Three distinct changes, solving three different problems. Worth separating, because most teams copy the first and skip the two that matter more.

1. Boundary versions instead of every combination

The matrix was reduced to boundary PHP versions, with redundant database combinations removed, while retaining full PHP version coverage. The announcement does not enumerate which specific versions and database pairings survived, so do not copy a version list from it. Copy the principle.

The principle: a full cross product of PHP versions and database versions tests combinations that cannot fail independently. If your code breaks on PHP 8.1, it breaks on PHP 8.1 against MySQL 8.0 and against MariaDB 10.6 and against every other engine in the grid. You paid for three jobs to learn one fact.

What genuinely varies by database is a much narrower set: SQL you wrote by hand, collation and charset behaviour, index limits, and strict mode. That is worth one or two dedicated jobs. It is not worth multiplying your entire suite by your engine count.

2. Build once, not once per job

The Gutenberg build was consolidated to run once per test run rather than once per job. In a twenty job matrix, an identical four minute build was executing twenty times to produce twenty byte-identical outputs.

This is the most common waste in WordPress plugin CI, and it is invisible in a per-job view because each individual job looks reasonable. It only looks absurd in aggregate.

3. Bounded Docker pull retries

Docker image pull retries were bounded. Unbounded retry against a flaky registry converts a fast failure into a slow one and burns runner minutes either way. Bounding it means a registry problem fails in a predictable window rather than sitting there.

The rerun rate is the real story

Sixty-eight percent of runs needing a rerun to pass means that for two runs out of three, a red result told you nothing. Under those conditions a developer stops reading failures. They hit rerun reflexively, because experience says the failure is probably noise. That reflex is how a genuine regression ships.

Cutting that to 36 percent does not make the suite trustworthy. It makes it twice as trustworthy as it was, which is the direction that matters. And notice how it was achieved: not by fixing individual flaky tests, but by removing infrastructure that had nothing to do with the assertions. Fewer jobs means fewer chances for a runner to fail. One build instead of twenty means nineteen fewer opportunities for a network hiccup during npm install.

Most flakiness in a WordPress plugin suite is not in the tests. It is in everything that has to succeed before the first assertion runs.

Measure before you cut

Start with the two numbers core reported: job count and job-minutes. On GitHub Actions the gh CLI gives you both.

Then the number that actually predicts developer trust, your rerun rate over the last fifty runs:

If that comes back above ten out of fifty, your suite has a credibility problem, and trimming the matrix will help more than writing new tests.

Rewriting a plugin matrix

Here is the shape most plugin workflows land in. Three PHP versions times three WordPress versions times two database engines is eighteen jobs.

Now apply boundaries. Test the oldest PHP you support and the newest, because failures cluster at the edges. Test the oldest WordPress you support and trunk. Give the alternate database one job rather than nine.

Eighteen jobs down to six. The middle-version job is the part people skip, and it is what keeps this honest. Pure boundary testing misses a regression that only affects PHP 8.2, which is rare but not hypothetical. One job costs little and closes that gap.

Note that fail-fast: false stays. With a trimmed matrix every job carries more information, so cancelling the rest on the first red result throws away more than it saves.

Build once, reuse everywhere

If your plugin has a build step, and anything with blocks or a bundled admin app does, split it into its own job and pass the output through as an artifact.

Two details that matter. Setting coverage: none on jobs that are not producing a coverage report removes Xdebug, and Xdebug makes PHPUnit dramatically slower. Run coverage in exactly one job. And retention-days: 1 keeps you from accumulating storage nobody will look at.

Bound your retries

Core bounded its Docker pull retries. The plugin equivalent is every network call in your setup: composer install, npm ci, and the WordPress core download that install-wp-tests.sh performs.

Three attempts with increasing backoff absorbs a transient registry blip. It does not hide a genuine outage for twenty minutes, and the log records that an attempt failed rather than silently succeeding on the second try. Silent retries are how a degrading dependency stays invisible until it fails completely.

Find the tests that are actually slow

Once the matrix is trimmed, remaining time is real test time. PHPUnit will tell you where it goes.

In a WordPress plugin suite the top of that list is almost always tests that hit the database when they did not need to, which is the same class of problem as a slow query in production and responds to the same diagnosis, or that call wp_remote_get() against a live host. The second category is also your flakiest, and it is worth fixing on correctness grounds alone. A unit test that makes a real HTTP request is not testing your code.

Wire that into your bootstrap and any test that was reaching the network becomes deterministic. If a test then fails, you have found a real dependency on a live service that was hiding in your suite.

Isolating a flaky test

Order dependence is the most common cause of a test that passes alone and fails in the suite. PHPUnit can prove it:

A test that passes with one seed and fails with another is leaking state. In WordPress that is usually an option, a transient, a registered post type, or a hook left attached. The core test case rolls back the database between tests, but it does not undo everything a plugin can touch.

What not to trim

Trimming has a failure mode, which is trimming until the suite is fast and meaningless. Four things stay regardless of budget.

Keep the oldest PHP version you claim to support in your readme.txt. If you advertise a floor, test the floor, or stop advertising it. Keep a job against WordPress trunk, because that is the only thing that warns you about the next release before your users do. Keep one job per genuinely distinct database engine, since collation and strict mode differences are real and will not surface anywhere else. And keep multisite coverage if you support multisite, because it is a different code path rather than a different environment.

Nothing else in a typical plugin matrix earns a full cross product.

Why the cross product felt safe

Worth understanding why these grids grow, because otherwise yours grows back in six months.

Nobody sits down and designs an eighteen job matrix. It accumulates. PHP 8.3 releases and somebody adds it. A user reports a MariaDB bug and somebody adds MariaDB. A support ticket comes in from a site on an old WordPress and somebody adds that version. Every single addition is individually justified, takes thirty seconds, and nobody ever removes anything, because removing a test feels like reducing safety.

The mistake in that reasoning is treating every cell as independent evidence. It is not. Adding MariaDB to a grid that already runs three PHP versions does not give you three new facts about MariaDB. It gives you one fact about MariaDB, three times, plus a 50 percent increase in the number of ways your CI can fail for reasons unrelated to your code.

The useful question for any matrix axis is: can this dimension fail independently of the others? PHP version and WordPress version can, because both change the API surface your code runs against. Database engine mostly cannot, outside of a narrow band of SQL and collation behaviour. Operating system almost never can, for a PHP plugin. Axes that cannot fail independently deserve one job, not a multiplier.

The cache layer most plugin CI gets wrong

Once the matrix is small, setup time dominates each remaining job. Two caches are worth configuring, and one common mistake is worth avoiding.

The PHP version belongs in the cache key. Composer resolves different packages for different PHP versions, so a shared key across your matrix will restore a cache built for the wrong one, and you get either a slow cache miss or, worse, subtly wrong dependencies.

The restore-keys fallback matters more than it looks. Without it, a one line change to composer.lock invalidates the cache completely and you download everything. With it, you restore the nearest previous cache and download only what changed.

The mistake to avoid: caching vendor/ directly rather than Composer’s cache directory. It appears faster and it produces confusing failures, because a restored vendor/ can contain packages that no longer match your lock file, and nothing in the run will tell you that is what happened.

While you are in there, check that your CI environment actually matches the environments you deploy to. The same reasoning that applies to separating dev, staging and production in wp-config.php applies to CI: a test environment that quietly differs from production is testing something other than what you ship.

If the rerun rate does not move

Trim the matrix, re-measure, and if reruns are still frequent, the flakiness is genuinely in your tests. At that point the triage order is worth following, because the categories have very different fix costs.

Start with time. Any test asserting on a timestamp, a date boundary, or an elapsed duration will fail eventually, usually at midnight UTC or across a month boundary. These are cheap to fix by freezing the clock or widening the assertion, and they are the most common single cause.

Next, ordering. Anything asserting on the order of a query result without an explicit ORDER BY is relying on the database to be consistent about something it never promised. The fix is in the query, not the test.

Then shared state, which the random seed technique above will find. Then genuine concurrency, which is rare in plugin suites and expensive to diagnose, so leave it until last.

Twenty consecutive passes in isolation, against failures inside the suite, is a definitive answer: the test is fine and something else is polluting it. That distinction saves hours, because the two cases have completely different fixes and they look identical in a CI log.

Where this meets the rest of your tooling

A trimmed matrix pairs with two habits worth having anyway.

The first is treating CI configuration as something you profile rather than something you accumulate. Most workflow files grow by addition. Somebody adds a PHP version when it releases, nobody removes the one that went end of life, and three years later you are running a grid nobody designed. Read your own workflow file once a quarter with the same eye you would bring to a slow query.

The second is separating the fast feedback loop from the thorough one. Linting and a single fast PHPUnit job belong on every push. The full matrix belongs on pull requests and on a schedule. Splitting them means the signal you actually read stays under two minutes, and the exhaustive run happens where waiting is acceptable.

Gating the matrix behind needs: quick also means a syntax error does not spend six jobs proving it is still a syntax error.

The checklist

• Record your current job count, job-minutes and rerun rate before changing anything. Without a baseline you cannot tell whether this worked.
• Collapse the PHP times database cross product to boundary versions plus one job per distinct engine.
• Add one middle-version job so a mid-range regression is not invisible.
• Move any build step into its own job and pass the result through as an artifact.
• Set coverage: none everywhere except the single job producing a coverage report.
• Bound every network retry to three attempts with backoff, and log each failed attempt.
• Profile the ten slowest tests, and stub outbound HTTP in the bootstrap.
• Run the suite with two random seeds to expose order dependence.
• Split the fast loop from the full matrix, and gate the matrix behind the fast job.
• Keep your advertised minimum PHP, a trunk job, one job per engine, and multisite if you support it.
• Re-measure. If the rerun rate did not move, the flakiness is in the tests and not the matrix.

What core demonstrated is that most CI cost is not test execution. It is duplicated setup, redundant combinations, and unbounded waiting on a network. You can usually take half of it out without losing a single assertion, and the suite gets more trustworthy in the process because there are fewer places for it to fail for reasons that have nothing to do with your code.

Start with the rerun rate. If your team has learned to hit the rerun button without reading the failure, that is the problem worth fixing first, and trimming the matrix is usually the cheapest way to fix it.