Hook

• Call back to the Atomicity Gap post. I asked the reader: "are you going to spend the time to properly crash test every possible boundary?" This post is me answering that question for Ductwork itself.
• Rhetorical question to the reader: how do you actually know your job backend survives a crash? Have you ever seen the test that proves it?
• The two ways crash tests usually get written, and why neither proves anything about a specific gap:
  • Fake the wreckage. Write the half-finished rows by hand, then check that recovery cleans them up. You're testing the cleanup, not the crash. Nothing proves the real process leaves behind exactly those rows.
  • Kill on a timer. Fork a real worker, wait a bit, SIGKILL it. Real crash, but it lands wherever the process happens to be. You never know which gap you tested, and you can't aim at a specific one.
• Bonus point: raise isn't a crash either. ensure runs, transactions roll back. SIGKILL skips all of that, so if something survives SIGKILL it survives an exception, but not the other way around.

Thesis: crash windows can be listed

• Durability isn't a vibe. Every multi-step state change is a finite list of gaps between statements, and you can name each one.
• At each gap you ask one question: if the process is SIGKILLed right here, what's left behind, and what recovers it?
• The answer is one of a short list: transaction rollback, the heartbeat reaper, a claim token that makes a stale write a no-op, a CAS that matches zero rows. Or nothing, and nothing is the bug.
• Then you write a test that kills a real process at exactly that line.
• Short version of the thesis line for the post: a durability guarantee is only as real as the crash you've actually tested.

The mechanism

• Show Ductwork::FaultInjection in full. It's 35 lines.
  • checkpoint(:after_branch_claim) sits in production code between two statements.
  • It's a no-op unless the DUCTWORK_FAULT env var names that checkpoint, e.g. after_branch_claim:kill.
  • Modes: kill (SIGKILL itself), sleep (hold the process at that line for a second), raise, exit.
  • Fun detail: no spec uses raise or exit. Every single one uses kill or sleep.
• The shape of every durability spec (walk through after_job_claim_spec.rb as the example):
  1. Set up rows the way they'd look mid-pipeline.
  2. Fork a real JobWorker / PipelineAdvancer with the fault set.
  3. It hits the checkpoint and kills itself. Assert termsig == KILL.
  4. Assert on the wreckage it actually left in the database. This is the part the "fake the wreckage" approach can't do.
  5. Run the reaper and a fresh process.
  6. Assert the work finished exactly once (or ran again exactly once, for at-least-once steps).
• sleep mode is the interesting one: it turns a race that normally lasts microseconds into a test where a second process gets to act inside the gap every time, on purpose.
• The "I want to be honest here" bit, quote or paraphrase the first commit message: "it still feels kinda gross to have points in the code where it's meant to blow up." Why it's worth it anyway: the checkpoint is a single ENV.key? check when the var isn't set, and the alternative is not knowing.
• Also honest: what the specs fake.
  • The heartbeat going stale: specs backdate last_heartbeat_at six minutes instead of waiting.
  • The reaper sweep: called directly with reap_all! instead of waiting for the supervisor.
  • One crash per test. No crash-during-recovery.

What we kill, and where

• The table. One row per checkpoint, OSS first, then Pro.

Checkpoint	Mode	What's left behind	What we assert
after_job_claim	kill	Job claimed, never ran	Reaper marks the execution crashed, a fresh one runs the job
during_job_execution	kill	Your code ran, result never recorded	Runs again exactly once (at-least-once), 2 executions
after_branch_claim	kill	Branch claimed, advancement left open	Reaper releases the branch, a fresh advancer finishes it
after_branch_advancement	kill	Advancement committed, process died after	Branch completes, still exactly 1 advancement (no double advance)
after_branch_candidate_select	sleep	Stale candidate held while another advancer moves the branch on	Chain advances exactly once, no duplicate step
before_advancement_create	sleep	Our process record reaped mid-claim	No orphaned advancement, process exits cleanly
during_expand_fanout (Pro)	kill	Some batches of children committed	Resumes from the next batch, no duplicate children
during_expand_succeeded (Pro)	kill	Payloads committed, winning execution never set	Retry's payloads become canonical, abandoned ones ignored
during_collapse_fanin (Pro)	kill	Siblings wired, fan-in barrier not bumped	Same collapse target reused, barrier bumped once

• One or two sentences pointing out the pattern in the "what's left behind" column: every row is a place where some of the work is durable and some isn't. That's the Atomicity Gap, just inside the framework instead of inside your job.
• Where the list comes from: a durability audit that walks every multi-step write and asks the SIGKILL question at each gap. TODO(Tyler): mention that the audit is an LLM skill (.claude/skills/audit-durability) or keep it generic? Could tie back to the Atomicity Gap audit prompt on the site.

The time a test found a different bug

• Story: the reaper race (c7efd16 then 7871753, July 2026).
• The worry from an audit: an advancer is in the middle of claiming a branch, its process record gets reaped (stale heartbeat), and the advancement gets created pointing at a process that no longer exists. That's an orphan no reaper sweep can ever find, so the branch is stuck forever.
• The test: before_advancement_create:sleep parks the advancer right before the insert, the parent force-reaps its process record, then lets it continue. Runs for real against Postgres.
• What it showed:
  • The orphan we were scared of never happens. Either the reaper goes second and sees the advancement, or it goes first and the insert fails on the foreign key and the whole claim rolls back.
  • But that foreign key failure raised ActiveRecord::InvalidForeignKey out of the claim and killed the advancer thread. Recovered eventually by the health check, but with no log saying why.
• The receipt: the first version of the spec had to rescue StandardError just so the forked process would exit instead of hanging the test. Show that snippet. The fix removed the rescue and added expect($?.exitstatus).to eq(0). Show the before/after diff.
• Takeaway: the test disproved the bug we were worried about and found one we hadn't thought of. Neither would show up by reading the code harder.
• Optional shorter second example: the stale branch candidate (a72d95e). Found by an audit at around 1M live branches, where the candidate scan got slow enough to open the window. after_branch_candidate_select:sleep turned that window into a test that fails every time. Be clear this one was pinned down by the test, not found by it.

How the other Ruby backends test this

• Keep this short and fair. The point isn't "they're bad," it's that this is uncommon, which is why it's worth writing about.
• Sidekiq (OSS): no process-kill tests in the suite. Crash recovery (super_fetch) lives in closed-source Pro, so its tests can't be checked.
• GoodJob: the README promises jobs are retried after SIGKILL. The spec for that builds the wreckage by hand (good_job.update!(performed_at: Time.current, finished_at: nil) in interrupt_errors_spec.rb) and checks the retry. The fake-the-wreckage approach from the hook.
• Solid Queue: the closest to us. Integration tests really SIGKILL forked workers and supervisors (forked_processes_lifecycle_test.rb). The kill is timed from outside, so it's the kill-on-a-timer approach. Also a different contract: the killed job gets marked failed with ProcessExitError, not re-run.
• Don't claim "we're the only ones who use real SIGKILL." Solid Queue does. The claim is: killing at a named line, a listed set of gaps, and sleep mode for races.
• Checked against main at: sidekiq f735096, goodjob 484e4af, solidqueue 73602fe. TODO: re-check right before publishing.
• Nod to prior art so nobody has to post it in the comments: named failure points aren't new in systems land (FreeBSD's fail(9), etcd's gofail, TiKV's fail-rs). It's just rare in Ruby job land.

The receipts

• Bring it home: go back to the promises from the Atomicity Gap post and map each one to rows in the table.
  • "Steps run at least once": during_job_execution, after_job_claim.
  • "No window between marking a step done and creating the next one": after_branch_advancement, after_branch_claim.
  • "Only one process can advance a step, so the chain can't fork": after_branch_candidate_select, before_advancement_create.
  • Pro's fan-out and fan-in don't duplicate or lose children: the three Pro rows.
• Link to the spec directory in the OSS repo so people can read them. TODO(Tyler): Pro specs are private, so describe them and link only the OSS ones?
• Call to action: steal the module. It's 35 lines, put a checkpoint between the Stripe call and the update! in your own scariest job and see what your backend actually does. Or let Ductwork be the thing that's already been crash tested.