Saga Compensations Under Partial Failure: Idempotent Undo Paths That Run

Saga Compensations Under Partial Failure: Idempotent Undo Paths That Run
14 views
Forward steps that partially commit are the failure mode that turns a tidy saga diagram into a 3 a.m. war room. The compensation you sketched on a whiteboard is useless if it is not idempotent, observable, and safe to retry after the orchestrator itself dies mid-undo. Seniors design the undo path with the same rigor as the happy path — keys, timeouts, and metrics included.
⚡ TL;DR: Persist saga state with step-level outcomes before acknowledging work. Make every compensation key-idempotent and reverse-ordered. Emit compensation success/failure metrics and alarm on stuck sagas. Pair with Idempotency Keys End-to-End and DynamoDB Streams Outbox.

Saga Compensations Under Partial Failure: production guidance

typescript
// saga/state.ts
export type StepStatus = "pending" | "committed" | "compensated" | "failed";

export interface SagaStep {
 name: string;
 status: StepStatus;
 forwardIdempotencyKey: string;
 compensateIdempotencyKey: string;
 resultRef?: string; // ARN / paymentId / bookingId
}

export async function markCommitted(sagaId: string, step: string, resultRef: string) {
 // ✅ Conditional write — never advance if already compensating
 await ddb.update({
 Key: { pk: `saga#${sagaId}` },
 ConditionExpression: "attribute_exists(pk) AND #st = :running",
 UpdateExpression:
 "SET steps.#n.#s = :c, steps.#n.resultRef = :r, updatedAt = :t",
 ExpressionAttributeNames: { "#n": step, "#s": "status", "#st": "status" },
 ExpressionAttributeValues: {
 ":c": "committed",
 ":r": resultRef,
 ":running": "running",
 ":t": Date.now(),
 },
 });
}
❌ Compensating from in-memory arrays after a crash — you will undo steps that never committed or skip the ones that did.

Compensations must be idempotent undo, not “delete if exists” folklore

typescript
// saga/compensate.ts
export async function compensatePayment(step: SagaStep) {
 if (!step.resultRef) return; // nothing committed
 await payments.refund({
 paymentId: step.resultRef,
 idempotencyKey: step.compensateIdempotencyKey, // ✅ stable forever
 reason: "saga_rollback",
 });
 // Refund API returns 200 on replay of same key — treat as success
}

export async function runCompensations(sagaId: string, steps: SagaStep[]) {
 // ✅ Reverse order of committed steps only
 for (const step of [...steps].reverse().filter((s) => s.status === "committed")) {
 try {
 await compensateDispatch[step.name](step);
 await markCompensated(sagaId, step.name);
 } catch (err) {
 await markCompensateFailed(sagaId, step.name, err);
 throw err; // leave remaining for redrive — do not invent success
 }
 }
}

Partial failure matrix

Forward outcome Compensation action
Never called Skip
Called, timeout, unknown Query side-effect by forward key; compensate if present
Committed Compensate with stable undo key
Compensate already done No-op (idempotent)
Compensate failed 3× Page; park saga in needs_manual
Same discipline as Exactly-Once Illusions — assume at-least-once for both forward and undo.

Observability that prevents a second outage

python
# metrics/saga_comp.py
from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="Sagas")

def emit_compensate(step: str, ok: bool, latency_ms: float):
 metrics.add_metric("CompensateLatencyMs", MetricUnit.Milliseconds, latency_ms)
 metrics.add_metric(
 "CompensateSuccess" if ok else "CompensateFailure",
 MetricUnit.Count,
 1,
 )
 metrics.add_dimension(name="step", value=step)
Alarm on CompensateFailure > 0 and on sagas older than 2 × max_step_sla still in running or compensating. Without those alarms, silent stuck bookings become customer-facing debt.

Closing checklist

✅ Dos – ✅ Persist step status before acknowledging the next hop – ✅ Use distinct forward and compensate idempotency keys – ✅ Compensate only committed steps, reverse order – ✅ Query-on-timeout when forward outcome is unknown – ✅ Alarm on stuck and failed compensations ❌ Don’ts – ❌ Don’t trust in-memory saga state across process death – ❌ Don’t make compensations “best effort” without redrive – ❌ Don’t reuse the forward key as the refund key – ❌ Don’t compensate steps that never left pending – ❌ Don’t hide compensate failures behind HTTP 200

Related reading

Last updated on September 11, 2026 Related: Lambda Timeouts, Retries, and DLQs: Idempotent Failure Handling; Multi-Agent Systems: How to Build Agent Teams That Do Not Hallucinate Each Other Into Failure; Docker Compose for Developers: Run Your Entire Stack Locally With One Command.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.