Skip to content

Interruptible Advancement

The expand transition fans out over a collection, creating a new step instance for every element. For modest collections this is no problem. But when the collection holds hundreds of thousands (or millions) of elements, materializing every step in one shot becomes a problem: one enormous database transaction and unbounded memory growth, with all of that progress lost if the process goes down partway through.

Ductwork Pro replaces this all-at-once expansion with interruptible advancement. The advancer materializes expanded steps in bounded batches, each committed in its own transaction. An uninterrupted advance works through every batch in one pass. Since each batch is saved as soon as it commits, an expansion that gets interrupted partway keeps the steps it already created and resumes from there. A single expand can handle millions of steps this way without long transactions or memory spikes.

Naive fan-out over a very large collection runs into hard limits:

  • Long transactions - inserting millions of rows in one transaction holds locks, bloats your WAL/undo logs, and risks timeouts
  • Memory pressure - building the full set of step records in memory at once can exhaust the advancer process
  • Unsafe shutdown - a crash, kill, or deploy arriving mid-expansion rolls back the whole transaction, discarding every step created so far

Interruptible advancement removes these limits, so large fan-outs (batch enrichment, per-record processing, large backfills) are something you can just do instead of something to work around.

When a pipeline reaches an expand transition, the Pro advancer materializes the expanded steps in batches rather than all at once:

  1. Batch creation - a bounded number of step instances are created and committed in a single transaction
  2. Wire and commit - each new step’s input is marked as wired in that same transaction, so once a batch commits it is never recreated
  3. Continue - the advance moves on to the next not-yet-wired batch, repeating until the collection is exhausted

A single uninterrupted advance completes the whole expansion in one pass. Splitting the work into separately committed batches keeps each transaction small and limits how many records are held in memory at once. Since each batch is saved as soon as it commits, it also means an interrupted expansion can be recovered (see Safe interruption).

You don’t need to change your pipeline definition. The same expand transition you already use gets this automatically:

class EnrichAllUsersDataPipeline < Ductwork::Pipeline
define do |pipeline|
pipeline.start(QueryUsersRequiringEnrichment)
.expand(to: LoadUserData)
end
end

If QueryUsersRequiringEnrichment returns ten elements or ten million, the advancer fans out the same way; the only difference is how many batches it takes to finish.

Because every batch commits independently, an expansion that’s interrupted partway doesn’t lose the steps it already created:

  • Interrupted mid-expansion - if the process crashes, is killed, or is forced down during a deploy (see Signal Handling), every batch that committed before the interruption survives. An in-flight batch that hadn’t committed yet rolls back cleanly, leaving no partial rows behind.
  • Resume on re-advance - when the branch is picked up again, expansion continues from the first batch that wasn’t yet wired. Already-created steps are not duplicated.

This makes large fan-outs safe to run across deploys and restarts. The work that committed survives the interruption, and the expansion picks up where it left off instead of starting over.

There’s nothing to configure. Interruptible advancement is always on for expand transitions in Ductwork Pro. The advancer handles batching on its own and there’s nothing to tune.

Interruptible advancement governs how expanded steps are created; it does not change the limit on how many may be created. The pipeline_advancer.steps.max_depth ceiling still applies across the full expansion. If a collection would push a branch past max_depth, the pipeline is halted just as it would be without batching. The limit is evaluated against the cumulative total, not per batch.

For very large fan-outs, raise or unset max_depth:

default: &default
pipeline_advancer:
steps:
max_depth: -1 # unlimited

If you’ve configured Metrics, a runaway fan-out still shows up:

  • pipeline.halted - incremented (tagged with the pipeline class) if an expansion exceeds the configured max_depth

Note that the batched fan-out bulk-inserts its steps, so it does not emit the per-step job.enqueued metric that ordinary transitions do. Instead, each committed batch is recorded in the logs as a Job batch enqueued entry. For a high-cardinality expand, watching the step count climb in the dashboard as batches commit is the easiest way to tell that a fan-out is making progress and hasn’t stalled.