An end-to-end Playwright suite that took hours on one machine, rebuilt as an eighty-way fanout on Azure. Three minutes wall clock, one merged report, and every piece of the infrastructure declared in Terraform.
Finning International sells and services Caterpillar equipment across three continents, and the portal its customers depend on had a real end-to-end suite: Playwright driving the actual application, the tests you want standing between a change and production. The problem was arithmetic. Run serially on one machine, the suite took hours, so it ran at night. A regression introduced at 10 a.m. was discovered the next morning, by whoever triaged the nightly run, long after everyone had stopped thinking about the change that caused it.
Slow suites do quiet damage: they make tests expensive, and people ration what’s expensive. Every new scenario made the night longer, so scenarios didn’t get written. Coverage sat at roughly a fifth of the use cases that mattered, not because anyone decided that was enough, but because the wall clock was doing the deciding.
Playwright will happily run --shard=3/80 and execute one eightieth of the suite. On a single machine that’s a party trick. The real question is coordination: how do eighty machines, all booting in the same second, each end up with a different shard number, no duplicates, no gaps, no coordinator database, no machine waiting on another?
The 8Examples habit applies to infrastructure as much as to products: walk the timeline concretely before building. What happens in the first second, when all eighty replicas ask at once? What happens when a claim crashes halfway? We drew the run from top to bottom until every arrow had an answer:
Notice how little is invented here. The fanout is a stock Azure Container App Job with parallelism = N. The slicing is Playwright’s own. The only thing we had to design was the small box on the left: a function that hands each replica a number, exactly once, under contention.
The whole coordination state is one blob: shards.json, a list of the shard numbers nobody has claimed yet. The Azure Function in front of it does one thing: take a lease on the blob, read the list and its ETag, remove one number, write the list back conditionally, release the lease, and return { yourShard: k }. No queue, no database, no leader election. A blob with a lease is already a lock, and an ETag is already a version check; Azure Storage was holding the primitives the whole time.
The lease matters for the bad day, not the good one. It expires on its own after sixty seconds, so a claim that dies mid-write can’t jam the line; the next replica walks over the corpse and takes the lock.
This is the moment we walked through most carefully on paper, because it’s the moment that would have been a production incident if we’d walked through it in production instead. Eighty claims arrive essentially at once. One invocation gets the lease; the other seventy-nine get 409 and back off with randomized exponential delays, so they don’t return as a synchronized stampede. And underneath the lease sits a second, paranoid layer: the write back to the blob carries If-Match with the ETag from the read. The lease is the lock; the ETag is proof the lock worked.
A sharded suite has a failure mode a serial suite doesn’t: a shard that silently vanishes takes its failures with it, and the report that remains looks green. So the runner script uploads its zipped blob report from an EXIT trap, which fires whether the tests passed, failed, or the process was killed. Failing tests still produce a report; that’s the point of them.
The merge step then downloads all N archives and runs playwright merge-reports into a single HTML report, indistinguishable from one machine having run everything. It exits zero only if all N reports exist and the merge succeeds. A missing shard fails the build loudly; nothing partial ever ships as a pass.
Every resource in the picture, the storage account, the container registry, the Container App environment and job, the Linux function app, Log Analytics and Application Insights wired through it, is declared in Terraform. Nothing was clicked into existence in the portal, which means nothing has to be remembered. The shard count itself is a variable; parallelism = 80 is a line in a file, and capacity planning is editing it.
That’s also the disaster recovery story, and it’s deliberately boring. The system holds no precious state; the blobs live one run at a time. If a region or a subscription disappears, recovery is terraform apply somewhere else and a pushed image: the same command that created the environment the first time, not a binder of steps nobody has rehearsed.
The local stack isn’t a toy version, it’s the same images, the same function, the same claim protocol, scaled with --scale runner=N. GitHub Actions runs the entire architecture on every push. When your disaster recovery drill and your CI run are the same artifact, the drill happens dozens of times a day.
The headline number is the wall clock: hours became three minutes. But the number that changed the engineering culture is the other one. Use-case coverage went from roughly 20% to 80%, and nobody mandated it. The price of a test collapsed; a new scenario adds an eightieth of its runtime to the wall clock, which rounds to nothing, so people wrote the tests they had always wanted to write.
And the suite moved. A run that costs three minutes doesn’t have to wait for nightfall; it runs on every push, and failures arrive while the change is still open in the editor, attributed to one commit instead of a day’s worth. The morning triage meeting quietly stopped existing.
Nothing in this system is exotic: a stock job runner, a seventy-line function, a blob, and Playwright’s own sharding. The work was walking the timeline first, eighty machines, second by second, until the race conditions had been found on paper instead of in production. What Finning bought wasn’t a faster suite. It was the ability to say yes, add the test without doing arithmetic about the night.