Engine overview

RatchetBatchEngine is the bundled batch execution engine. It ships inside the main ratchet package (src/core/batch/engine/) — there is no separate install, no optional dynamic import, and no activation step. When batch apply or a headless verb runs, the engine is constructed and called in-process.

The engine's contract is one forced transition per call: it spawns exactly one agent for the chosen transition, maps the session to a structured result, and returns. The autonomous loop that calls it repeatedly is the apply-batch skill, not the CLI.

Architecture at a glance

Both entry surfaces converge on the change-scoped core, which spawns a single agent through the SWE-ReX runtime and maps the session to a StepResult. Only runStep (the batch-facing surface) takes the lock and derives the transition; the headless verbs reuse runChangeStep directly with a forced transition.

The two entry surfaces

runStep — batch-facing

runStep(context: ResolvedStepContext): Promise<StepResult>

Called by ratchet batch apply (src/commands/batch/apply.ts). Its responsibilities before delegating to the core:

1. Acquires the per-batch single-flight lock (see Lock).
2. Re-derives the authoritative transition from on-disk state via computeNextTransition, overriding the coarse hint in context.transition.
3. Checks for an unresolved park and returns without spawning if one is present.
4. Adapts ResolvedStepContext into a ChangeStepContext (with the derived transition) and delegates the spawn-and-map body to runChangeStep.

ResolvedStepContext requires a batch field; the lock, transition derivation, and park-precedence are deliberately runStep's concern and stay outside the core so the headless verbs can reuse runChangeStep directly.

runChangeStep — change-scoped core

runChangeStep(ctx: ChangeStepContext): Promise<StepResult>

The shared core both runStep and the headless verbs invoke. It does not acquire the per-batch lock and does not call computeNextTransition — the transition in ctx.transition is treated as forced and final. See Change-step core for the full interface and behavior.

Headless verbs (ratchet propose, ratchet apply, ratchet verify — in src/commands/propose.ts and via src/commands/change-step-common.ts) call runChangeStep directly with a ChangeStepContext that has no batch, forcing exactly the verb's own transition. Run state is kept change-locally under .ratchet/changes/<change>/.run/. See Standalone settings for how settings are resolved without a manifest.

runDecompositionStep — phase-scoped

runDecompositionStep(context: DecompositionStepContext): Promise<StepResult>

Called by ratchet batch apply when the next runnable step is a reachable, ungated phase whose changes list is still empty (a decomposition step, not a change step). Unlike the change path it is keyed off the phase, not a change: it carries no change and no transition. Its responsibilities mirror runStep but author a phase's intents rather than advance a change:

1. Acquires the per-batch single-flight lock (same lock as runStep).
2. Guarantees the canonical decomposition command (decompose-phase) is present in the spawn locus (same render-or-fail discipline as the per-change transitions — see Skill in spawn locus).
3. Builds decomposition instructions that delegate to the canonical decompose-phase skill (/rct:decompose-phase <phase>, resolved per agent), injecting the empty phase's goal/success/proof-of-work and the prior phases' shipped results as context (delegated-lifecycle: the engine orchestrates the spawn; the skill authors the intents).
4. Spawns exactly one agent through the same runtime selection, streaming/rendering, and journal-delta snapshot the change path uses.
5. Maps the session to a StepResult (transition: 'decompose'). It never calls computeNextTransition and never authors batch.yaml itself — the skill writes the phase's concrete change intents.

A decomposition has no change, so its journal/park state is keyed by the phase name (the decomposition agent reports with ratchet batch report <batch> --change <phase> ...). Once the intents are authored, the next batch apply selects the phase's first ready change as an ordinary propose/apply/verify step — the loop continues with no manual stop/propose/resume detour (#30).

Single-step contract

batch apply advances exactly one transition per invocation. No internal loop exists in the CLI or the engine. The caller — typically the apply-batch skill — is responsible for invoking batch apply repeatedly until the batch completes.

Transition derivation

computeNextTransition (src/core/batch/engine/transition.ts) reads the live change directory to decide the authoritative next transition:

On-disk state	Next transition
No change directory	propose
Change directory exists, no plan.md	propose
plan.md present, not all task checkboxes checked	apply
All task checkboxes checked, no verify completion in journal	verify
Archived, or verify completion already in journal	undefined (nothing runnable)

"All task checkboxes checked" is tested by counting - [ ] / - [x] lines in plan.md; every task must be checked (tasksComplete === tasksTotal > 0).

runStep calls computeNextTransition after acquiring the lock and uses the result as the authoritative transition, falling back to context.transition only when the function returns undefined (i.e., the change is already done). runChangeStep never calls it.

The single journal-aware done-rule

"Done" has one definition, computed in one place (hasJournaledVerify / isChangeDone in src/core/batch/engine/transition.ts) and honored uniformly by status derivation (computeBatchStatus), step selection (pickNextStep in src/core/batch/engine/selection.ts), and computeNextTransition. A change is done only when its plan tasks are all checked and the run journal carries a completion entry for the verify transition — or the change is archived.

The in-between state — tasks all checked but no journaled verify — is the derived awaiting-verify status (ChangeStatus in src/core/batch/status.ts). It is explicitly NOT done: status reports awaiting-verify, and verify is the next runnable transition, so verify actually runs as a gate before a change is done rather than being skipped on task-checkboxes alone.

Step selection

pickNextStep (src/core/batch/engine/selection.ts) is the single selection engine: the one implementation of runnable-step selection in the codebase. batch apply calls it over computeBatchStatus and the manifest phases to pick the next step. Status derivation and step selection share one runnable-change eligibility walk (firstRunnableChange in the same module): computeBatchStatus calls it to derive the change-level next; pickNextStep's change branch calls it and layers the boundary-proof interposition on the picked phase. One set, one walk — status and selection agree by construction.

type ApplyTarget =
  | { kind: 'change'; phase: Phase; change: string; changeDone: string }
  | { kind: 'decompose'; phase: Phase }
  | { kind: 'proof-of-work'; phase: Phase }
  | { kind: 'pr'; phase: Phase; boundary?: PrGroupBoundary };

Selection proceeds in load-bearing branch order (each branch returns its target or undefined, and the first non-undefined wins):

1. change / boundary proof-of-work — firstRunnableChange finds the first ungated phase's first change whose derived status is ready, in-progress, or awaiting-verify. Before returning that change, the immediately-preceding phase P's one-time boundary proof-of-work is interposed (see Phase gates and proof-of-work).
2. decompose — when no ungated change is runnable, a reachable, ungated phase whose changes list is empty is surfaced as a decomposition step (from computeBatchStatus.next, which sets decompose only once no change-level next exists — so a still-gated empty phase is never picked). Its predecessor's boundary proof runs first, exactly as before a change step.
3. terminal proof-of-work — once every change is done and nothing is left to decompose, the LAST phase's unrun boundary proof is surfaced (from computeBatchStatus.next.proof) — the last phase has no successor, so its proof is selected here and run/recorded rather than at a later boundary.
4. whole-batch PR — at genuine batch completion under prGrouping: whole-batch with no PR already opened, a pr target for the terminal phase.
5. stacked PR tail — at genuine batch completion under a STACKED grouping mode (per-phase/per-change), a pr target for the first unopened group boundary (each subsequent apply opens the next group, in boundary order).

Because done is the single journal-aware predicate, an awaiting-verify change (tasks all checked, no journaled verify) is done: false and is therefore selectable — firstRunnableChange includes awaiting-verify in RUNNABLE_STATUSES, so selection schedules its verify transition as the gate that must run before it can be done, rather than skipping it on task-checkboxes alone. computeBatchStatus derives the same change as next via the same walk, so status and selection schedule the same verify gate.

Before returning a runnable change in phase Q, pickNextStep interposes the immediately-preceding phase P's proof-of-work as a boundary step: P is done (else Q would be gated), so when P has no recorded proof outcome yet, pickNextStep returns a { kind: 'proof-of-work'; phase: P } target before Q's change. The set of already-recorded phases is passed in (built from readProofOfWorkByPhase), so the boundary runs once. Once P's proof is recorded, the next apply consults the recorded verdict: a passing proof (or warn) advances into Q, while a failing hard-gate proof leaves Q gated and pickNextStep returns no Q change — batchApplyCommand's no-step branch then cites P's failing proof. The first phase has no predecessor, so it yields no proof step. See Phase gates and proof-of-work.

When no runnable step is found, pickNextStep returns undefined (no target). This mirrors the batch-done rule in batch status: status and selection key off the same two facts — "phase decomposed?" (changes.length > 0) and "phase reachable?" (ungated) — via the shared firstRunnableChange walk, so they cannot disagree about whether a reachable empty phase is outstanding work.

Batch status and the decomposition step

A multi-phase batch is done only once every reachable phase is decomposed, all its changes are done, AND the terminal phase's boundary proof-of-work is recorded as satisfied. The old done arithmetic counted only declared change intents (doneCount === changeCount), so a later phase with an empty changes list contributed zero and the batch flipped done the moment the first phase's changes finished — even though the later phase had no concrete intents yet (#30). computeBatchStatus now folds in whether any reachable (ungated) phase is still undecomposed: such a phase keeps the batch in-progress and is surfaced as next (a decomposition step carrying the phase, no change).

Terminal-phase proof gate. Each phase's boundary proof-of-work runs when entering the next phase, but the last phase has no successor — so its proof would never run, and the batch would report done without it. computeBatchStatus closes that hole: once every change is done and nothing is left to decompose, the terminal phase's proof must be recorded as satisfied (gatePassed: true) for the batch to be done. When the terminal proof has not run, status stays in-progress and surfaces it as next ({ phase, proof: true }, no change); pickNextStep selects it, and batch apply runs and records it. A failing terminal hard-gate proof keeps the batch out of done with nothing auto-runnable — batch apply's no-step output cites the failing proof and the operator must batch rerun-proof (or fix the cause). This also means a single-phase batch gates on its one phase's proof before reporting done.

batch apply drives the decomposition natively: when pickNextStep surfaces a decomposition step it calls runDecompositionStep, which spawns one agent that delegates to the canonical decompose-phase skill to author the phase's concrete change intents into batch.yaml from the prior phases' shipped results. The next batch apply then selects the phase's first ready change — no manual stop/propose/resume detour (#30).

A still-gated empty phase (its prior phase has unfinished work) is NOT surfaced as a decomposition step yet — the unfinished prior-phase change is selected first.

Phase gates and proof-of-work

Phase boundaries are enforced through two independent mechanisms:

Gate modes

The gate field in BatchSettings (set in the batch manifest or project config) controls when the engine parks for human approval:

Mode	Behavior
voluntary	Never parks for approval automatically.
after-propose	Parks for approval after each completed propose transition, before apply.
every-phase	Parks for approval after every completed change transition (propose, apply, and verify). Decomposition and PR-open steps never park for approval under any gate.
autonomous	Never parks for approval; agent blockers still park.

Under after-propose, a completed propose transition causes runStep / runChangeStep to return awaiting-approval instead of advanced. Under every-phase, every completed change transition (propose, apply, and verify) parks the same way. The step does not re-run until the park is cleared. voluntary and autonomous never park for approval (autonomous still parks on agent blockers). The decision is made by the pure parksForApproval(gate, transition) matrix (src/core/batch/engine/approval-gate.ts), the single source of truth threaded through shouldParkForApproval and the decomposition and PR-open step call sites — so a decomposition or PR-open step never parks for approval under any gate (a decomposition's authored intents are reviewed when each change's propose parks; a PR is itself the human checkpoint).

Proof-of-work

Each phase in the manifest carries a proofOfWork definition. The engine exposes runProofOfWork (src/core/batch/engine/proof-of-work.ts) for running it once all changes in a phase are done:

Kind	Behavior
integration	Runs a bash command; evaluates a pass condition against exit status and stdout.
blackbox	Same execution path as integration.

llm-judge is not yet supported by batch apply. It is a recognized proof-of-work kind in the schema, but no judge is wired, so a manifest with an llm-judge proof-of-work is rejected at validation (ratchet validate <batch> --type batch and the apply load path both fail) with: "llm-judge proof-of-work is not yet supported by batch apply; use integration or blackbox." Use integration or blackbox.

Pass conditions (for integration/blackbox):

Condition string	Passes when
"" / exit 0 / exit-zero / exit code 0	Command exits 0.
leading exit-zero directive (e.g. exit code 0 — new tests pass, exit-zero: suite green)	Command exits 0. A condition that begins with an exit 0 / exit-zero / exit code 0 directive — optionally followed by punctuation/prose — gates on the exit status and is not substring-matched against stdout.
contains:<text>	Command exits 0 and stdout contains <text>.
regex:<pattern>	Command exits 0 and stdout matches the regex.
anything else (not an exit-code directive)	Treated as substring: command exits 0 and stdout contains the string.

Gate policy (ProofOfWorkPolicy):

Policy	Behavior
hard-gate (default)	A failed proof blocks the phase and prevents the next phase from starting.
warn	A failed proof is recorded but the phase is allowed to complete.

gatePassed is true when the proof passed, or when the policy is warn.

Execution at the phase boundary

batch apply is runProofOfWork's live caller. When phase P's changes are all done and the next reachable phase Q still has outstanding work, the host loop (pickNextStep in src/core/batch/engine/selection.ts, driven by batchApplyCommand in src/commands/batch/apply.ts) runs P's proof-of-work at the boundary before entering Q. The terminal phase has no successor Q, so its proof runs at a different trigger: once every change in the batch is done and nothing is left to decompose, the terminal phase's proof is surfaced and run the same way — it is what holds the batch out of done until it is recorded as satisfied (see Terminal-phase proof gate). The predecessor's boundary proof also runs before a decomposition step: an undecomposed phase is entered off its predecessor's shipped slice, so the predecessor's proof runs first, exactly as it does before a change step.

The executed command is the phase's configured proofOfWork.run, run in the project root — ratchet injects no package manager, test runner, or command string of its own. The boundary check runs at most once per boundary: the verdict is journaled as a proof-of-work entry carrying a ProofOfWorkRecord, and the next batch apply reads the recorded set (readProofOfWorkByPhase). The verdict therefore survives across the stateless single-step apply invocations. See Run-state locus.

A recorded verdict is not permanent. When a proof was recorded FAILED for a fixable cause (a misconfigured pass, a flaky run, an env fix), the operator runs ratchet batch rerun-proof [name] --phase P (see commands: batch rerun-proof). It appends an append-only proof-of-work-invalidated marker that supersedes the record — the original entry is left in place for the audit trail. Because the single record fold (proofRecordsFromEntries) treats the marker as removing P from the record map, the next batch apply sees P as not-recorded (the REC decision re-reads "no"), re-runs P's configured boundary proof, and records a fresh verdict that re-derives the gate. No journal surgery, no second gate path.

The recorded verdict drives the phase gate. computeBatchStatus derives Q's gate from P's recorded gatePassed: a failing hard-gate proof (gatePassed: false) keeps Q blocked with a gatedBy report citing P's failing proof and its detail, while a passing proof — or any warn verdict, which the recorder always stores as gatePassed: true — opens Q. The single selection engine (pickNextStep, via the shared firstRunnableChange walk that skips gated phases) reads that single derived gate, so what status reports blocked is exactly what selection refuses to run. Under warn the failure is surfaced when the boundary proof runs (rendered as ⚠ failed (warn)) but never blocks progression.

Outcomes

StepResult (public contract)

type StepState =
  | 'advanced'
  | 'blocked'
  | 'awaiting-approval'
  | 'nothing-ready';

interface StepResult {
  state: StepState;
  change: string;          // the decomposed phase's name on a decomposition step
  transition: StepKind;    // 'propose' | 'apply' | 'verify' | 'decompose'
  blocker?: string;         // present when state is 'blocked'
  approvalRequest?: string; // present when state is 'awaiting-approval'
  journalRefs?: number[];   // indices of journal entries this step wrote
  message?: string;
}

State	Meaning
advanced	The transition completed; the agent reported a completion journal entry CORROBORATED against the on-disk change state (see Session-to-outcome mapping).
blocked	The step requires attention: the agent raised a blocker, the agent crashed, an internal failed state was mapped here, or a claimed --complete did not corroborate against disk (see below). The step is resumable.
awaiting-approval	A completed change transition (corroborated) parked under an approval gate: propose under after-propose, or propose/apply/verify under every-phase. Parked until approved or feedback is recorded; the park message names the transition that completed.
nothing-ready	No runnable step exists (all done, all gated, or all blocked/parked).

EngineStepOutcome (internal)

The engine computes an EngineStepOutcome (in src/core/batch/engine/context.ts) with an additional internal state failed. The toStepResult function maps failed → blocked before returning the public StepResult, so a crashed or non-zero agent surfaces as blocked (keeping the batch resumable) rather than a clean advance.

Session-to-outcome mapping

mapSessionToOutcome (src/core/batch/engine/outcome.ts) examines the journal entries the agent wrote during the session, the process exit status, AND the on-disk change state it already snapshots (diskEvidence: { before, after }) to corroborate a claimed completion:

1. A blocker or needs-input journal entry → blocked.
2. Non-zero exit without a completion → failed (surfaces as blocked). When the transition's (or batch-driven pr / phase-decomposition step's) resolved spec explicitly named a model AND the agent wrote zero journal entries during the session (the argv-rejection signature) AND the spawn exited with a real non-zero exit code (signal === null, not a signal kill), the failure carries a model-failure attribution hint — naming the stage, agent, exact model string, and supplying scope (the project config vs the batch manifest), phrased as "if this model id is invalid…" guidance. The hint opens the detail field above the captured stderr tail (for --json consumers) AND is threaded into the surfaced blocker/message fields, so every human-facing rendered surface that prints them carries it: the non-JSON batch apply blocked line, the parked-step reason shown on resume, the journal entry message recorded for the transition, and the standalone change-step renderer. The pr stage spawn is attributed via its pr stage entry's supplying scope; the phase-decomposition spawn is likewise attributed via its own decompose stage entry's supplying scope. The hint never interprets stderr content and never diagnoses. A signal-killed spawn (e.g. a timeout SIGKILL, OOM kill — exitCode: null, signal: 'SIGKILL') under a valid explicit model is NOT a real exit code, so the hint is suppressed there even with zero journal entries; the bare-failure fallback (describeExit naming the signal, stderr tail intact) surfaces the failure on every rendered surface without the hint, so an externally killed agent under a perfectly valid model never misdirects the operator into "fixing" a model id that was never the problem. A bare-name spec, a stage-map-driven decompose spawn (default agent, no model), a scope-less (standalone) path, or a failure after journal progress surfaces byte-for-byte today's output — only the argv-rejection attribution branch changes.
3. A completion journal entry, CORROBORATED against diskEvidence before advancing (only propose | apply | verify are corroborated; decompose and pr step kinds key off synthetic journal keys with no change directory and stay byte-for-byte today's behavior):
   • Mismatch — the disk disagrees with the claim → blocked, blocker Reported complete but disk disagrees: <reason>.
     • propose → after-state must have the change directory AND a plan.md (after.exists && after.hasPlan; absolute, so a resumed propose over an existing directory corroborates).
     • apply → after-state is fully applied (after.applied) OR at least one task was checked off this session (after.tasksComplete > before.tasksComplete).
     • verify → after-state is applied AND the completion message carries the verification verdict (matches the exported VERIFY_VERDICT_PATTERN, the canonical Final Assessment shapes the rct:verify workflow template authors: "Ready for archive" or "N critical issue(s) found…"). The pattern checks verdict PRESENCE, not polarity — a critical-issue verdict is still a verdict; gating done on a passing verdict is out of scope. Verify produces no disk artifact, so its evidence is (a) the change is actually applied and (b) the message carries the verdict. An archived after-state reports applied: true, so a late archival cannot false-block.
   • Completion + non-zero exit/signal → blocked, blocker Agent reported completion but exited <describeExit> — the reported work may be incomplete; review and resume. (fail-closed; the manifest allows "at least warned, arguably blocked" — this engine picks blocked).
   • Under an approval gate → awaiting-approval (the gate×transition matrix decides which transitions park: propose under after-propose; propose/apply/verify under every-phase; only a CORROBORATED completion parks here — a mismatch blocks first; the park message names the transition that completed, e.g. "Apply complete; awaiting approval.").
   • Otherwise → advanced.
4. Zero exit without a completion → blocked; on-disk evidence (plan.md appeared, task checkboxes advanced) is surfaced in the message but the step never auto-advances on unreported work.

The branch order in the completion path is: reported blocker (1) → non-zero exit without completion (2) → corroboration mismatch (3.a) → completion + crash (3.b) → approval park (3.c) → advanced (3.d) → zero-exit no-report blocked (4). The mismatch precedes the crash check (a disk mismatch is the more specific, actionable evidence) and both precede the approval park so an uncorroborated propose can no longer park as awaiting-approval.

Journal, run-state, and lock

Journal and state

The engine is resumable across crashes. Two files live at the run-state locus (see Run-state locus):

File	Contents
journal.jsonl	Append-only log of agent reports (progress, blocker, needs-input, completion) and user answers/feedback. Read tolerantly: a partial trailing line from a mid-crash write is silently dropped.
state.json	Currently parked steps (blocked / awaiting-approval) for resume.

The locus is:

Step kind	Run directory
Batch step	.ratchet/batches/<batch>/run/
Standalone change step (headless verbs)	.ratchet/changes/<change>/.run/

Lock

runStep guards each batch step with an exclusive per-batch lock file at .ratchet/batches/<batch>/run/step.lock. The lock holds the owning pid and at timestamp. A stale lock left by a dead process (pid no longer alive) is reclaimed automatically. Concurrent calls from live processes throw BatchLockedError. The lock is local-host only; it is not NFS- or multi-host-safe.

Headless verbs call runChangeStep directly and do not acquire this lock.

One batch apply tick end-to-end

ratchet batch apply <name>
│
├─ 1. Load manifest + batch settings
├─ 2. computeBatchStatus → pickNextStep → select first ready/in-progress/
│      awaiting-verify change (skips gated phases; picks first change with deps
│      done and not parked) — an awaiting-verify change schedules its verify gate.
│      If no change is runnable but a reachable phase has empty `changes`,
│      pickNextStep returns a DECOMPOSE target → engine.runDecompositionStep
│      (spawns one agent delegating to the decompose-phase skill), then exit
│
├─ 3. Pre-check park (CLI): if unresolved blocked/awaiting-approval → print hint, exit
│
├─ 4. Build ResolvedStepContext (coarse transition hint from computeNextTransition)
│
└─ 5. engine.runStep(context)
       │
       ├─ 5a. Acquire .ratchet/batches/<batch>/run/step.lock
       │
       ├─ 5b. computeNextTransition → authoritative transition from disk
       │
       ├─ 5c. Check park (engine): if unresolved → return blocked StepResult (no spawn)
       │
       └─ 5d. runChangeStep(changeStepContext)
               │
               ├─ Resolve locus: { batch } → .ratchet/batches/<batch>/run/
               ├─ Build agent instructions (transition, phase, change, guidance, resume)
               ├─ Resolve runtime (local → ReX sidecar; docker → ReX sidecar;
               │  remote → RexRemoteRuntime) — see Agent runtime
               ├─ Spawn ONE agent; stream output live
               ├─ Snapshot journal delta + on-disk state delta
               ├─ mapSessionToOutcome → EngineStepOutcome
               ├─ Stamp .ratchet.yaml metadata (propose only)
               ├─ Append outcome journal entry at locus
               └─ toStepResult → StepResult

       └─ 5e. Release lock

├─ 6. persistStepOutcome: parkStep / clearParkedStep in state.json
└─ 7. Render result (text or --json)