Day 75: Autoscaling Retrieval and Embed Jobs

Re-embedding the world on one giant ECS service will melt your vector DB and your wallet. Autoscale embed/retrieve workers with queues, concurrency caps, and backpressure so ingest cannot outrun index health.

⚡ TL;DR: SQS → workers (ECS/Glue) with max concurrency tied to vector DB write capacity. Separate query path from ingest. Shed load with 429s, not silent corruption.

Topology

docs → ingest API → SQS → embed workers → vector DB
user queries → retrieve API → (read replicas / separate pool)

# worker.py
import os, time, boto3
sqs = boto3.client("sqs")
MAX_INFLIGHT = int(os.environ.get("MAX_INFLIGHT", "8"))

def loop(queue_url: str):
    inflight = 0
    while True:
        if inflight >= MAX_INFLIGHT:
            time.sleep(0.2)
            continue
        msgs = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=1).get("Messages", [])
        for m in msgs:
            inflight += 1
            try:
                embed_and_upsert(m)
                sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=m["ReceiptHandle"])
            finally:
                inflight -= 1

Autoscale on queue depth, not CPU alone

# target tracking on ApproximateNumberOfMessagesVisible
TargetValue: 50

Cap desired count so workers * writes_per_sec < db_safe_write_rps.

Closing checklist

• [ ] Queue-based ingest
• [ ] Concurrency caps from DB capacity
• [ ] Separate query vs ingest pools
• [ ] DLQ for poison docs
• [ ] Backpressure to API (429)

Series navigation

Day 74: Batch Inference Overnight · Day 76: p99 of Agents: Queueing, Not Just Model Latency

Last updated September 11, 2026