multi-agent failure modes: Swarm Failure Modes

Day 55: Swarm Failure Modes
8 views
This guide focuses on multi-agent failure modes for production systems, with practical trade-offs for reliability, security, and cost. Swarms fail in boring ways: two planners emit incompatible plans, two implementers open duplicate PRs, and nobody notices until review. Treat split-brain and duplication as first-class failure modes with detectors, not postmortems.
⚡ TL;DR: One active plan per run_id. Lease the implementer slot. Fingerprint PR titles/branches. Alert on divergent plan hashes for the same ticket.

multi-agent failure modes: production guidance

python
# detect/split_brain.py
def assert_single_plan(store, ticket_id: str, run_id: str):
    plans = store.list_plans(ticket_id)
    active = [p for p in plans if p.status == "active"]
    if len(active) > 1:
        raise RuntimeError(f"split_brain:{ticket_id}:{[p.hash for p in active]}")
    if active and active[0].run_id != run_id:
        raise RuntimeError("stale_run_holds_plan")

Duplicate PR detector

typescript
// detect/dupPr.ts
export function prFingerprint(title: string, files: string[]): string {
  const norm = title.toLowerCase().replace(/\s+/g, " ").trim();
  const f = [...files].sort().join("|");
  return `${norm}::${f}`;
}

export function findDupes(open: { title: string; files: string[]; url: string }[]) {
  const map = new Map<string, string[]>();
  for (const p of open) {
    const k = prFingerprint(p.title, p.files);
    map.set(k, [...(map.get(k) || []), p.url]);
  }
  return [...map.values()].filter((v) => v.length > 1);
}
✅ Close or mark duplicates automatically; never merge both.

Leases for implementers

python
# lease.py
def acquire_implementer(redis, run_id: str, ttl=600) -> bool:
    # ✅ SET NX EX — only one implementer mutates the branch
    return bool(redis.set(f"lease:impl:{run_id}", "1", nx=True, ex=ttl))

Closing checklist

  • [ ] Single active plan per ticket/run
  • [ ] Implementer leases
  • [ ] Duplicate PR fingerprints in CI
  • [ ] Alarm on multiple active plans
  • [ ] Cancel stale runs when a newer plan activates

Series navigation

Day 54: Specialist Tools per Agent · Day 56: Supervisor Patterns on Bedrock Agents Last updated September 11, 2026 Related: Multi-Agent Systems: How to Build Agent Teams That Do Not Hallucinate Each Other Into Failure; Day 51: Multi-Agent Roles: Planner, Implementer, Critic; Day 59: Multi-Agent Eval: Task Suites, Not Vibes.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.