Cutting CI Feedback Time with Checkpoint Restarts

Context

At a fintech building real-time trading UIs, small teams ship often. The whole delivery rhythm depends on how quickly a change gets from a commit to a trustworthy green tick.

Problem

Our CI/CD pipelines regularly took more than ten minutes. That is long enough for people to context-switch, stack up more changes, and stop treating the pipeline as a conversation. And when a long-running job failed partway through, it started again from zero, so one failure could double the wait.

What I did

• Optimised the pipelines so a change got its answer in under two minutes instead of over ten.
• Made long-running jobs restartable from checkpoints. Instead of one all-or-nothing step, the work records its progress, and a re-run carries on from the last completed checkpoint rather than from the start. It’s the same idea as Resumable Batch Migrations: idempotent units, a durable record of what’s done, safe re-runs.

Outcome

Pipelines went from over ten minutes to under two, roughly a 5× improvement in feedback time, and a failure in a long job no longer meant paying for the whole run again.

Patterns that help (general, not a log of that pipeline)

• Measure the wait, not the jobs. What matters is the time until a developer gets an answer they can act on; the fastest, most-likely-to-fail checks should report first (Shift Left Testing).
• Independent jobs in parallel, cached work never repeated. The usual first wins on any slow pipeline.
• Treat every long job as a batch job. If it can fail halfway, it should be able to resume halfway.
• Fail loudly and specifically. A checkpoint is only useful if the failure tells you where you are; transient errors can retry, real ones should stop clearly.

What I’d do differently

Track pipeline duration and flake rate as first-class metrics from the start (DORA Metrics). Slow pipelines creep in one reasonable-looking step at a time, and a trend line catches that long before people start complaining.

Related notes

Fast Feedback from Deploy Previews · A Testing Strategy Teams Actually Adopted · Slicing Pull Requests · Playwright