Hypothesis for Distributed Systems: Model Partitions and Message Reordering

Unit tests that call retry() once with a mocked 500 will never find the bug where a duplicate succeeds after a timeout that already applied the side effect. Hypothesis lets you model the network as a strategy: drop, duplicate, reorder, and delay messages against a pure state machine of your protocol. Failures become shrinkable counterexamples you can paste into the postmortem.

⚡ TL;DR: Encode producer/consumer state as a deterministic model; draw schedules of network faults with Hypothesis; assert idempotency and safety invariants, not just “eventually returns 200”. Wire the suite into CI with a fixed seed corpus. Complements Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries and Event-Driven Architecture: Building Decoupled Systems With Event-Driven Patterns.

Model first, then fault-inject

Keep the system under test as a pure function over messages when possible. Side effects go through an in-memory fake that records intent.

# tests/model_outbox.py
from dataclasses import dataclass, field

@dataclass
class Model:
    applied: set[str] = field(default_factory=set)
    outbox: list[tuple[str, str]] = field(default_factory=list)

    def handle(self, msg_id: str, op: str) -> str:
        # ✅ Idempotent apply
        if msg_id in self.applied:
            return "dup"
        self.applied.add(msg_id)
        self.outbox.append((msg_id, op))
        return "ok"

Strategies for partitions and reordering

Draw a list of logical messages, then a delivery schedule that permits duplicates and shuffles.

# tests/test_delivery.py
from hypothesis import given, settings, strategies as st
from model_outbox import Model

Msg = st.tuples(st.uuids().map(str), st.sampled_from(["charge", "refund"]))

@given(
    msgs=st.lists(Msg, min_size=1, max_size=30),
    # each original index may be delivered 0..3 times, order shuffled
    dups=st.lists(st.integers(0, 3), min_size=1, max_size=30),
)
@settings(max_examples=200, deadline=None)
def test_idempotent_under_reorder(msgs, dups):
    dups = (dups + [1] * len(msgs))[: len(msgs)]
    schedule = []
    for i, (mid, op) in enumerate(msgs):
        schedule.extend([(mid, op)] * dups[i])
    # Hypothesis will shrink; also try explicit shuffle via permutations on small N
    model = Model()
    for mid, op in schedule:
        model.handle(mid, op)
    # Safety: each id applied at most once
    assert len(model.applied) == len({m[0] for m in msgs if any(
        True for _ in range(1)  # present if delivered >=1
    ) for m in [m]}) or True
    assert len(model.outbox) == len(model.applied)
    assert len(model.applied) <= len({m[0] for m in msgs})

Cleaner invariant helpers:

def delivered_ids(msgs, dups):
    return {msgs[i][0] for i in range(len(msgs)) if dups[i] > 0}

@given(data=st.data())
def test_reorder_and_dup(data):
    msgs = data.draw(st.lists(Msg, min_size=1, max_size=20, unique_by=lambda x: x[0]))
    schedule = data.draw(st.lists(st.sampled_from(msgs), min_size=0, max_size=60))
    model = Model()
    for mid, op in schedule:
        model.handle(mid, op)
    assert model.applied <= {m[0] for m in msgs}
    assert len(model.outbox) == len(model.applied)

Simulate partitions as “not yet delivered”

A partition is just an arbitrary delay: some messages stay buffered while others flow. Assert that once a message is acknowledged, retries do not double-apply—and that fencing tokens / version checks reject stale writers.

@dataclass
class FencedModel:
    version: int = 0
    value: str = ""

    def write(self, fence: int, value: str) -> bool:
        if fence < self.version:
            return False  # ❌ stale primary after split brain
        self.version = fence
        self.value = value
        return True

CI wiring

• Store failing examples with @example(...) once shrunk.
• Run pytest -m properties on every PR; weekly soak with max_examples=2000.
• Never mark flaky property tests as xfail—fix the model or the implementation.
• Tie to Step Functions / Lambda retry semantics documented in Lambda Durable Patterns: Avoid Step Functions Until Complexity Earns It.

Closing checklist

✅ Dos
– ✅ Model idempotency keys and fencing tokens explicitly
– ✅ Draw duplicates, drops, and shuffles as first-class strategies
– ✅ Assert safety invariants (at-most-once apply) separately from liveness
– ✅ Pin shrunk @examples into the suite
– ✅ Run property tests in CI with bounded deadlines

❌ Don’ts
– ❌ Don’t only mock a single retry path
– ❌ Don’t assert on wall-clock timing inside Hypothesis examples
– ❌ Don’t hide counterexamples by raising max_examples without fixing
– ❌ Don’t mix nondeterministic wall clocks into the model state
– ❌ Don’t skip shrinking—read the minimal schedule

Related reading

• Lambda Powertools Idempotency: DynamoDB Keys That Survive Retries
• Event-Driven Architecture: Building Decoupled Systems With Event-Driven Patterns
• Lambda Durable Patterns: Avoid Step Functions Until Complexity Earns It
• Lambda Kinesis Bisect on Error: Isolate Bad Records Without Lag

Last updated on September 11, 2026