DynamoDB Hot Key Mitigation: Write Sharding With Scatter-Gather Reads

One celebrity partition key will throttle an otherwise healthy table long before you hit account limits. Write sharding spreads load across N physical keys; scatter-gather reads stitch them back. The senior move is choosing N, bounding gather fan-out, and accepting that absolute counters become approximate unless you add a rollup path.

⚡ TL;DR: Append a calculated shard suffix to hot keys (pk#shard = hash(id) % N). Write to one shard; read with bounded parallel GetItem/Query. Cap N by p99 gather latency. Pair with Multi-Tenant Rate Limits and Lambda Reserved Concurrency Bulkheads.

Detect the hot key before customers do

# ops/hot_key.py
# CloudWatch Contributor Insights / CloudWatch metrics — illustrative
THRESH_WCU = 800  # per partition soft alarm under on-demand

def is_hot(partition_key: str, consumed_wcu: float) -> bool:
    return consumed_wcu >= THRESH_WCU  # ✅ alert + auto-recommend shard count

❌ Waiting for ProvisionedThroughputExceeded in user-facing APIs — by then you are already shedding traffic.

Calculated write shards

// ddb/shard.ts
import { createHash } from "node:crypto";

export function shardOf(id: string, n: number): number {
  const h = createHash("sha256").update(id).digest();
  return h.readUInt32BE(0) % n; // ✅ stable for a given N
}

export function shardedPk(base: string, id: string, n: number): string {
  return `${base}#${shardOf(id, n)}`;
}

export async function incrCounter(tenantId: string, delta: number, n = 16) {
  // For pure counters, shard by random/round-robin instead of id hash:
  const shard = Math.floor(Math.random() * n);
  await ddb.update({
    Key: { pk: `tenant#${tenantId}#cnt#${shard}`, sk: "total" },
    UpdateExpression: "ADD #c :d",
    ExpressionAttributeNames: { "#c": "count" },
    ExpressionAttributeValues: { ":d": delta },
  });
}

Use random/RR shards for write-heavy counters; use hash(id) % N when each entity must land on one shard for point reads.

Scatter-gather with a latency budget

// ddb/gather.ts
export async function gatherCount(tenantId: string, n = 16, budgetMs = 40) {
  const started = Date.now();
  const keys = Array.from({ length: n }, (_, i) => ({
    pk: `tenant#${tenantId}#cnt#${i}`,
    sk: "total",
  }));
  // ✅ Parallel GetItem with AbortSignal budget
  const parts = await Promise.all(
    keys.map((Key) =>
      ddb.get({ Key, AbortSignal: AbortSignal.timeout(budgetMs) }).catch(() => null)
    )
  );
  const sum = parts.reduce((a, p) => a + (p?.Item?.count ?? 0), 0);
  metrics.timing("GatherMs", Date.now() - started);
  return { sum, partial: parts.some((p) => p === null) };
}

Rollups beat giant gathers

For dashboards, maintain an async rollup via Streams → Lambda that writes tenant#id#cnt#rollup. Clients read the rollup; writers keep sharding. Same outbox mindset as DynamoDB Streams Outbox.

Closing checklist

✅ Dos
– ✅ Alarm on per-partition WCU/RCU before hard throttles
– ✅ Pick shard function deliberately (hash vs random)
– ✅ Bound gather parallelism and time budget
– ✅ Expose partial results instead of lying
– ✅ Add rollups when N grows past ~16

❌ Don’ts
– ❌ Don’t shard every key “just in case”
– ❌ Don’t gather 256 keys on the user path
– ❌ Don’t change N without a dual-read migration plan
– ❌ Don’t ignore skew inside shards (tenant-of-tenants)
– ❌ Don’t treat approximate counters as financial truth

Related reading

• Multi-Tenant Rate Limits: Redis Cluster Token Buckets Without Hot Keys
• Lambda Reserved Concurrency Bulkheads That Protect Tenant Workloads
• DynamoDB Streams Outbox: Domain Events Without Dual-Write Failures
• Read-Your-Writes on Global Tables

Last updated on September 11, 2026