Saga Compensations Under Partial Failure: Idempotent Undo Paths That Run

Forward steps that partially commit are the failure mode that turns a tidy saga diagram into a 3 a.m. war room. The compensation you sketched on a whiteboard is useless if it is not idempotent, observable, and safe to retry after the orchestrator itself dies mid-undo. Seniors design the undo path with the same rigor as the happy path — keys, timeouts, and metrics included.

⚡ TL;DR: Persist saga state with step-level outcomes before acknowledging work. Make every compensation key-idempotent and reverse-ordered. Emit compensation success/failure metrics and alarm on stuck sagas. Pair with Idempotency Keys End-to-End and DynamoDB Streams Outbox.

Persist step outcomes before the next hop

// saga/state.ts
export type StepStatus = "pending" | "committed" | "compensated" | "failed";

export interface SagaStep {
  name: string;
  status: StepStatus;
  forwardIdempotencyKey: string;
  compensateIdempotencyKey: string;
  resultRef?: string; // ARN / paymentId / bookingId
}

export async function markCommitted(sagaId: string, step: string, resultRef: string) {
  // ✅ Conditional write — never advance if already compensating
  await ddb.update({
    Key: { pk: `saga#${sagaId}` },
    ConditionExpression: "attribute_exists(pk) AND #st = :running",
    UpdateExpression:
      "SET steps.#n.#s = :c, steps.#n.resultRef = :r, updatedAt = :t",
    ExpressionAttributeNames: { "#n": step, "#s": "status", "#st": "status" },
    ExpressionAttributeValues: {
      ":c": "committed",
      ":r": resultRef,
      ":running": "running",
      ":t": Date.now(),
    },
  });
}

❌ Compensating from in-memory arrays after a crash — you will undo steps that never committed or skip the ones that did.

Compensations must be idempotent undo, not “delete if exists” folklore

// saga/compensate.ts
export async function compensatePayment(step: SagaStep) {
  if (!step.resultRef) return; // nothing committed
  await payments.refund({
    paymentId: step.resultRef,
    idempotencyKey: step.compensateIdempotencyKey, // ✅ stable forever
    reason: "saga_rollback",
  });
  // Refund API returns 200 on replay of same key — treat as success
}

export async function runCompensations(sagaId: string, steps: SagaStep[]) {
  // ✅ Reverse order of committed steps only
  for (const step of [...steps].reverse().filter((s) => s.status === "committed")) {
    try {
      await compensateDispatch[step.name](step);
      await markCompensated(sagaId, step.name);
    } catch (err) {
      await markCompensateFailed(sagaId, step.name, err);
      throw err; // leave remaining for redrive — do not invent success
    }
  }
}

Partial failure matrix

Same discipline as Exactly-Once Illusions — assume at-least-once for both forward and undo.

Observability that prevents a second outage

# metrics/saga_comp.py
from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="Sagas")

def emit_compensate(step: str, ok: bool, latency_ms: float):
    metrics.add_metric("CompensateLatencyMs", MetricUnit.Milliseconds, latency_ms)
    metrics.add_metric(
        "CompensateSuccess" if ok else "CompensateFailure",
        MetricUnit.Count,
        1,
    )
    metrics.add_dimension(name="step", value=step)

Alarm on CompensateFailure > 0 and on sagas older than 2 × max_step_sla still in running or compensating. Without those alarms, silent stuck bookings become customer-facing debt.

Closing checklist

✅ Dos
– ✅ Persist step status before acknowledging the next hop
– ✅ Use distinct forward and compensate idempotency keys
– ✅ Compensate only committed steps, reverse order
– ✅ Query-on-timeout when forward outcome is unknown
– ✅ Alarm on stuck and failed compensations

❌ Don’ts
– ❌ Don’t trust in-memory saga state across process death
– ❌ Don’t make compensations “best effort” without redrive
– ❌ Don’t reuse the forward key as the refund key
– ❌ Don’t compensate steps that never left pending
– ❌ Don’t hide compensate failures behind HTTP 200

Related reading

• Idempotency Keys End-to-End: API Gateway Through Step Functions Safely
• DynamoDB Streams Outbox: Domain Events Without Dual-Write Failures
• Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers
• Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries

Last updated on September 11, 2026