14 views
Forward steps that partially commit are the failure mode that turns a tidy saga diagram into a 3 a.m. war room. The compensation you sketched on a whiteboard is useless if it is not idempotent, observable, and safe to retry after the orchestrator itself dies mid-undo. Seniors design the undo path with the same rigor as the happy path — keys, timeouts, and metrics included.
⚡ TL;DR: Persist saga state with step-level outcomes before acknowledging work. Make every compensation key-idempotent and reverse-ordered. Emit compensation success/failure metrics and alarm on stuck sagas. Pair with Idempotency Keys End-to-End and DynamoDB Streams Outbox.
Saga Compensations Under Partial Failure: production guidance
typescript
❌ Compensating from in-memory arrays after a crash — you will undo steps that never committed or skip the ones that did.
// saga/state.ts
export type StepStatus = "pending" | "committed" | "compensated" | "failed";
export interface SagaStep {
name: string;
status: StepStatus;
forwardIdempotencyKey: string;
compensateIdempotencyKey: string;
resultRef?: string; // ARN / paymentId / bookingId
}
export async function markCommitted(sagaId: string, step: string, resultRef: string) {
// ✅ Conditional write — never advance if already compensating
await ddb.update({
Key: { pk: `saga#${sagaId}` },
ConditionExpression: "attribute_exists(pk) AND #st = :running",
UpdateExpression:
"SET steps.#n.#s = :c, steps.#n.resultRef = :r, updatedAt = :t",
ExpressionAttributeNames: { "#n": step, "#s": "status", "#st": "status" },
ExpressionAttributeValues: {
":c": "committed",
":r": resultRef,
":running": "running",
":t": Date.now(),
},
});
}
Compensations must be idempotent undo, not “delete if exists” folklore
typescript
// saga/compensate.ts
export async function compensatePayment(step: SagaStep) {
if (!step.resultRef) return; // nothing committed
await payments.refund({
paymentId: step.resultRef,
idempotencyKey: step.compensateIdempotencyKey, // ✅ stable forever
reason: "saga_rollback",
});
// Refund API returns 200 on replay of same key — treat as success
}
export async function runCompensations(sagaId: string, steps: SagaStep[]) {
// ✅ Reverse order of committed steps only
for (const step of [...steps].reverse().filter((s) => s.status === "committed")) {
try {
await compensateDispatch[step.name](step);
await markCompensated(sagaId, step.name);
} catch (err) {
await markCompensateFailed(sagaId, step.name, err);
throw err; // leave remaining for redrive — do not invent success
}
}
}
Partial failure matrix
| Forward outcome | Compensation action |
|---|---|
| Never called | Skip |
| Called, timeout, unknown | Query side-effect by forward key; compensate if present |
| Committed | Compensate with stable undo key |
| Compensate already done | No-op (idempotent) |
| Compensate failed 3× | Page; park saga in needs_manual |
Observability that prevents a second outage
python
Alarm on # metrics/saga_comp.py
from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit
metrics = Metrics(namespace="Sagas")
def emit_compensate(step: str, ok: bool, latency_ms: float):
metrics.add_metric("CompensateLatencyMs", MetricUnit.Milliseconds, latency_ms)
metrics.add_metric(
"CompensateSuccess" if ok else "CompensateFailure",
MetricUnit.Count,
1,
)
metrics.add_dimension(name="step", value=step)
CompensateFailure > 0 and on sagas older than 2 × max_step_sla still in running or compensating. Without those alarms, silent stuck bookings become customer-facing debt.
Closing checklist
✅ Dos – ✅ Persist step status before acknowledging the next hop – ✅ Use distinct forward and compensate idempotency keys – ✅ Compensate only committed steps, reverse order – ✅ Query-on-timeout when forward outcome is unknown – ✅ Alarm on stuck and failed compensations ❌ Don’ts – ❌ Don’t trust in-memory saga state across process death – ❌ Don’t make compensations “best effort” without redrive – ❌ Don’t reuse the forward key as the refund key – ❌ Don’t compensate steps that never leftpending
– ❌ Don’t hide compensate failures behind HTTP 200
Related reading
- Idempotency Keys End-to-End: API Gateway Through Step Functions Safely
- DynamoDB Streams Outbox: Domain Events Without Dual-Write Failures
- Exactly-Once Illusions: Design At-Least-Once Plus Truly Idempotent Handlers
- Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
