Hypothesis for Distributed Systems: Model Partitions and Message Reordering

Hypothesis for Distributed Systems: Model Partitions and Message Reordering
3 views
Unit tests that call retry() once with a mocked 500 will never find the bug where a duplicate succeeds after a timeout that already applied the side effect. Hypothesis lets you model the network as a strategy: drop, duplicate, reorder, and delay messages against a pure state machine of your protocol. Failures become shrinkable counterexamples you can paste into the postmortem.
⚡ TL;DR: Encode producer/consumer state as a deterministic model; draw schedules of network faults with Hypothesis; assert idempotency and safety invariants, not just “eventually returns 200”. Wire the suite into CI with a fixed seed corpus. Complements Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries and Event-Driven Architecture: Building Decoupled Systems With Event-Driven Patterns.

Hypothesis for Distributed Systems: production guidance

Keep the system under test as a pure function over messages when possible. Side effects go through an in-memory fake that records intent.
python
# tests/model_outbox.py
from dataclasses import dataclass, field

@dataclass
class Model:
 applied: set[str] = field(default_factory=set)
 outbox: list[tuple[str, str]] = field(default_factory=list)

 def handle(self, msg_id: str, op: str) -> str:
 # ✅ Idempotent apply
 if msg_id in self.applied:
 return "dup"
 self.applied.add(msg_id)
 self.outbox.append((msg_id, op))
 return "ok"

Strategies for partitions and reordering

Draw a list of logical messages, then a delivery schedule that permits duplicates and shuffles.
python
# tests/test_delivery.py
from hypothesis import given, settings, strategies as st
from model_outbox import Model

Msg = st.tuples(st.uuids().map(str), st.sampled_from(["charge", "refund"]))

@given(
 msgs=st.lists(Msg, min_size=1, max_size=30),
 # each original index may be delivered 0..3 times, order shuffled
 dups=st.lists(st.integers(0, 3), min_size=1, max_size=30),
)
@settings(max_examples=200, deadline=None)
def test_idempotent_under_reorder(msgs, dups):
 dups = (dups + [1] * len(msgs))[: len(msgs)]
 schedule = []
 for i, (mid, op) in enumerate(msgs):
 schedule.extend([(mid, op)] * dups[i])
 # Hypothesis will shrink; also try explicit shuffle via permutations on small N
 model = Model()
 for mid, op in schedule:
 model.handle(mid, op)
 # Safety: each id applied at most once
 assert len(model.applied) == len({m[0] for m in msgs if any(
 True for _ in range(1) # present if delivered >=1
 ) for m in [m]}) or True
 assert len(model.outbox) == len(model.applied)
 assert len(model.applied) <= len({m[0] for m in msgs})
Cleaner invariant helpers:
python
def delivered_ids(msgs, dups):
 return {msgs[i][0] for i in range(len(msgs)) if dups[i] > 0}

@given(data=st.data())
def test_reorder_and_dup(data):
 msgs = data.draw(st.lists(Msg, min_size=1, max_size=20, unique_by=lambda x: x[0]))
 schedule = data.draw(st.lists(st.sampled_from(msgs), min_size=0, max_size=60))
 model = Model()
 for mid, op in schedule:
 model.handle(mid, op)
 assert model.applied <= {m[0] for m in msgs}
 assert len(model.outbox) == len(model.applied)

Simulate partitions as “not yet delivered”

A partition is just an arbitrary delay: some messages stay buffered while others flow. Assert that once a message is acknowledged, retries do not double-apply—and that fencing tokens / version checks reject stale writers.
python
@dataclass
class FencedModel:
 version: int = 0
 value: str = ""

 def write(self, fence: int, value: str) -> bool:
 if fence < self.version:
 return False # ❌ stale primary after split brain
 self.version = fence
 self.value = value
 return True

CI wiring

Closing checklist

✅ Dos – ✅ Model idempotency keys and fencing tokens explicitly – ✅ Draw duplicates, drops, and shuffles as first-class strategies – ✅ Assert safety invariants (at-most-once apply) separately from liveness – ✅ Pin shrunk @examples into the suite – ✅ Run property tests in CI with bounded deadlines ❌ Don’ts – ❌ Don’t only mock a single retry path – ❌ Don’t assert on wall-clock timing inside Hypothesis examples – ❌ Don’t hide counterexamples by raising max_examples without fixing – ❌ Don’t mix nondeterministic wall clocks into the model state – ❌ Don’t skip shrinking—read the minimal schedule

Related reading

Last updated on September 11, 2026 Related: Multi-Agent Systems: How to Build Agent Teams That Do Not Hallucinate Each Other Into Failure; Multi-Agent Systems: Orchestrating Claude, GPT-4o, and Specialized Models Together; Multi-Model Routing: Cheap Draft Models Plus Expensive Verifiers.

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.