Posted inAI AWS Machine Learning

OpenSearch Serverless: Semantic Scratch Memory for Multi-Turn Coding Agents

No Comments
18 views
ElastiCache Redis is excellent for exact scratchpads and tool-result caches (Redis scratchpads). It is a poor fit when the planner asks “did we already fail a similar refactor on auth middleware?” across fuzzy wording and prior sessions. OpenSearch Serverless (vector search collection) is the unfair advantage for semantic scratch memory: embed summaries of tool outcomes, retrieve top-k by meaning, keep IAM and network inside AWS, skip cluster babysitting. Pair with DynamoDB session META for cursors and Redis for hot exact keys — this post is the semantic recall layer.
⚡ TL;DR: Create an OpenSearch Serverless vector collection; index {session_id, tenant_id, text, embedding, tool_name, ts} for each notable tool outcome. At plan time, k-NN retrieve similar failures/successes and inject into the prompt under a strict token budget. Related: Redis scratchpads, Bedrock Converse toolConfig, DynamoDB Streams session expiry.

Memory tiers for coding agents

Tier Store Query Lifetime
Working scratch Redis Exact key Minutes–hours
Session cursor / ledger DynamoDB PK/SK Session + TTL
Semantic recall OpenSearch Serverless k-NN + filters Hours–days
Long-term knowledge RAG index / docs Hybrid Weeks+
Do not dump full file contents into vectors on every keystroke — embed summaries (what changed, test result, error signature). Full blobs stay in S3 with conditional writes.

Collection and index shape

Use a vector search collection type. Network policy: private VPC access from agent Lambdas only. Encryption: AWS-owned or CMK. Data access policy: least-privilege IAM principal for the indexer and retriever roles.
json
{
  "settings": { "index": { "knn": true } },
  "mappings": {
    "properties": {
      "tenant_id": { "type": "keyword" },
      "session_id": { "type": "keyword" },
      "tool_name": { "type": "keyword" },
      "outcome": { "type": "keyword" },
      "summary": { "type": "text" },
      "created_at": { "type": "date" },
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "nmslib",
          "space_type": "cosinesimil",
          "parameters": { "ef_construction": 128, "m": 16 }
        }
      }
    }
  }
}
Pick embedding dimension to match your model (Titan Embeddings, Cohere on Bedrock, etc.). Filter by tenant_id on every query — semantic bleed across tenants is a security bug, not a recall feature.

Index on tool outcome

typescript
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
// Use official OpenSearch client against the Serverless collection endpoint

const bedrock = new BedrockRuntimeClient({});

async function embed(text: string): Promise<number[]> {
  const res = await bedrock.send(
    new InvokeModelCommand({
      modelId: "amazon.titan-embed-text-v2:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({ inputText: text.slice(0, 8000) }),
    })
  );
  const payload = JSON.parse(new TextDecoder().decode(res.body));
  return payload.embedding;
}

export async function indexToolMemory(row: {
  tenantId: string;
  sessionId: string;
  toolName: string;
  outcome: "ok" | "fail";
  summary: string;
}) {
  const vector = await embed(row.summary);
  await osClient.index({
    index: "agent-scratch-memory",
    body: {
      tenant_id: row.tenantId,
      session_id: row.sessionId,
      tool_name: row.toolName,
      outcome: row.outcome,
      summary: row.summary,
      created_at: new Date().toISOString(),
      embedding: vector,
    },
  });
}
typescript
// ❌ Embedding raw 200kb stack traces and entire file diffs
await indexToolMemory({ summary: hugeDiff });
// Noise + cost + prompt pollution; summarize first
Summaries should look like: "typecheck failed on src/auth/middleware.ts — Cannot find name 'UserContext'; after rename refactor".

Retrieve at plan time

typescript
export async function recallSimilar(opts: {
  tenantId: string;
  query: string;
  k?: number;
}) {
  const vector = await embed(opts.query);
  const result = await osClient.search({
    index: "agent-scratch-memory",
    body: {
      size: opts.k ?? 5,
      query: {
        bool: {
          filter: [{ term: { tenant_id: opts.tenantId } }],
          must: [
            {
              knn: {
                embedding: { vector, k: opts.k ?? 5 },
              },
            },
          ],
        },
      },
      _source: ["summary", "tool_name", "outcome", "created_at", "session_id"],
    },
  });
  return result.body.hits.hits.map((h: any) => h._source);
}
Inject into the system/developer message as a bounded bullet list. Cap tokens hard — semantic memory that steals half the context window is a regression. Prefer failures over successes when ranking for refactor tasks.

Ops: TTL, cost, and failure modes

OpenSearch Serverless bills on OCU (indexing + search). For scratch memory:
  • Time-based deletion: daily job deletes created_at < now-7d per tenant, or use ISM policies where supported.
  • Session end: on DynamoDB session expiry (Streams hooks), optionally delete-by-query session_id.
  • Cold start / scaling: Serverless removes node mgmt; still watch OCU minimums — tiny hobby traffic can feel pricey vs a shared Redis-only design.
  • Fallback: if OS is down, degrade to Redis exact cache + no semantic recall; do not block the agent loop.
Symptom Cause Fix
Cross-tenant recall Missing tenant filter Mandatory filter + integration test
Irrelevant hits Embedding raw dumps Better summaries; outcome filter
Latency spike Large k + huge _source k≤5; store summary only
Cost surprise Always-on OCUs + chatty index Batch index; sample successes

Production checklist

  • [ ] Vector collection private; data-access policy scoped to indexer/retriever roles.
  • [ ] Every query filters tenant_id (automated test).
  • [ ] Index summaries, not raw diffs; S3 holds blobs.
  • [ ] Prompt injection of recalls is token-budgeted.
  • [ ] TTL/ISM or sweeper for 3–14 day scratch horizon.
  • [ ] Degrade path if OpenSearch errors (Redis-only).
  • [ ] Embed model id pinned in SSM; dimension matches mapping.
  • [ ] Complement — not replace — Redis exact scratchpads.
Semantic scratch memory makes multi-turn coding agents feel like they “remember last Tuesday’s failed auth refactor” — without pretending Redis GET is a vector database.

Prompt packing without poisoning the planner

Bad recall injection looks like dumping five paragraphs of prior failures into the system prompt. Good packing:
text
## Prior similar attempts (tenant-scoped, last 7d)
- [fail][typecheck] src/auth/middleware.ts — missing UserContext after rename (session s_0918)
- [fail][apply_patch] patch rejected — import cycle auth↔users (session s_0919)
- [ok][run_tests] fixed by introducing shared types package (session s_0920)
Use these as hints; verify against current tree before re-applying patches.
Rules:
  1. Max 3–5 bullets; each ≤ 200 characters.
  2. Prefer outcome=fail for repair tasks; mix for greenfield.
  3. Never include secrets, tokens, or raw env dumps in summaries at index time.
  4. Cite session_id so humans can drill into DynamoDB/S3 artifacts.
  5. If cosine scores are weak (below your threshold), inject nothing — silence beats noise.
A/B this with eval traces in Athena — measure pass rate with vs without recall.

IAM sketch for indexer vs retriever

Split roles so a read-only planner path cannot delete the index:
json
{
  "Effect": "Allow",
  "Action": ["aoss:APIAccessAll"],
  "Resource": "arn:aws:aoss:us-east-1:123456789012:collection/colid"
}
Tighten with collection data-access policies: indexer Write/CreateIndex, retriever Read/DescribeCollectionItems only. Attach VPC endpoint policies so only the agent subnets reach the collection endpoint. Rotate nothing in env files — use task roles (KMS grants mindset).

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Most viewed

  1. Python Decorators Explained: From Simple Wrappers to Production Patterns233 views
  2. Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three105 views
  3. AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API87 views
  4. Python String Methods: Every str Method With Real Production Examples72 views
  5. REST API Design Best Practices: The Patterns That Make APIs a Joy to Use64 views

Newly added

  1. n8n vs AI Agent: I Gave Both the Same Job (Local, Ollama)
  2. Local AI Video Generator: Build Your Own App (Free, Python, No GPU)
  3. AI Wrote Raw Binary Instead of Code. Here’s What Happened (WebAssembly)
  4. I Built a Browser That Only Runs Encrypted Binary (Ep 1)
  5. RAG From Scratch: Chunking, Embeddings, Reranking and Evaluation (Real Numbers)

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.

Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.