OpenSearch Serverless: Semantic Scratch Memory for Multi-Turn Coding Agents

ElastiCache Redis is excellent for exact scratchpads and tool-result caches (Redis scratchpads). It is a poor fit when the planner asks “did we already fail a similar refactor on auth middleware?” across fuzzy wording and prior sessions. OpenSearch Serverless (vector search collection) is the unfair advantage for semantic scratch memory: embed summaries of tool outcomes, retrieve top-k by meaning, keep IAM and network inside AWS, skip cluster babysitting. Pair with DynamoDB session META for cursors and Redis for hot exact keys — this post is the semantic recall layer.

⚡ TL;DR: Create an OpenSearch Serverless vector collection; index {session_id, tenant_id, text, embedding, tool_name, ts} for each notable tool outcome. At plan time, k-NN retrieve similar failures/successes and inject into the prompt under a strict token budget. Related: Redis scratchpads, Bedrock Converse toolConfig, DynamoDB Streams session expiry.

Memory tiers for coding agents

Do not dump full file contents into vectors on every keystroke — embed summaries (what changed, test result, error signature). Full blobs stay in S3 with conditional writes.

Collection and index shape

Use a vector search collection type. Network policy: private VPC access from agent Lambdas only. Encryption: AWS-owned or CMK. Data access policy: least-privilege IAM principal for the indexer and retriever roles.

{
  "settings": { "index": { "knn": true } },
  "mappings": {
    "properties": {
      "tenant_id": { "type": "keyword" },
      "session_id": { "type": "keyword" },
      "tool_name": { "type": "keyword" },
      "outcome": { "type": "keyword" },
      "summary": { "type": "text" },
      "created_at": { "type": "date" },
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "nmslib",
          "space_type": "cosinesimil",
          "parameters": { "ef_construction": 128, "m": 16 }
        }
      }
    }
  }
}

Pick embedding dimension to match your model (Titan Embeddings, Cohere on Bedrock, etc.). Filter by tenant_id on every query — semantic bleed across tenants is a security bug, not a recall feature.

Index on tool outcome

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
// Use official OpenSearch client against the Serverless collection endpoint

const bedrock = new BedrockRuntimeClient({});

async function embed(text: string): Promise<number[]> {
  const res = await bedrock.send(
    new InvokeModelCommand({
      modelId: "amazon.titan-embed-text-v2:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({ inputText: text.slice(0, 8000) }),
    })
  );
  const payload = JSON.parse(new TextDecoder().decode(res.body));
  return payload.embedding;
}

export async function indexToolMemory(row: {
  tenantId: string;
  sessionId: string;
  toolName: string;
  outcome: "ok" | "fail";
  summary: string;
}) {
  const vector = await embed(row.summary);
  await osClient.index({
    index: "agent-scratch-memory",
    body: {
      tenant_id: row.tenantId,
      session_id: row.sessionId,
      tool_name: row.toolName,
      outcome: row.outcome,
      summary: row.summary,
      created_at: new Date().toISOString(),
      embedding: vector,
    },
  });
}

// ❌ Embedding raw 200kb stack traces and entire file diffs
await indexToolMemory({ summary: hugeDiff });
// Noise + cost + prompt pollution; summarize first

Summaries should look like: "typecheck failed on src/auth/middleware.ts — Cannot find name 'UserContext'; after rename refactor".

Retrieve at plan time

export async function recallSimilar(opts: {
  tenantId: string;
  query: string;
  k?: number;
}) {
  const vector = await embed(opts.query);
  const result = await osClient.search({
    index: "agent-scratch-memory",
    body: {
      size: opts.k ?? 5,
      query: {
        bool: {
          filter: [{ term: { tenant_id: opts.tenantId } }],
          must: [
            {
              knn: {
                embedding: { vector, k: opts.k ?? 5 },
              },
            },
          ],
        },
      },
      _source: ["summary", "tool_name", "outcome", "created_at", "session_id"],
    },
  });
  return result.body.hits.hits.map((h: any) => h._source);
}

Inject into the system/developer message as a bounded bullet list. Cap tokens hard — semantic memory that steals half the context window is a regression. Prefer failures over successes when ranking for refactor tasks.

Ops: TTL, cost, and failure modes

OpenSearch Serverless bills on OCU (indexing + search). For scratch memory:

• Time-based deletion: daily job deletes created_at < now-7d per tenant, or use ISM policies where supported.
• Session end: on DynamoDB session expiry (Streams hooks), optionally delete-by-query session_id.
• Cold start / scaling: Serverless removes node mgmt; still watch OCU minimums — tiny hobby traffic can feel pricey vs a shared Redis-only design.
• Fallback: if OS is down, degrade to Redis exact cache + no semantic recall; do not block the agent loop.

Production checklist

• [ ] Vector collection private; data-access policy scoped to indexer/retriever roles.
• [ ] Every query filters tenant_id (automated test).
• [ ] Index summaries, not raw diffs; S3 holds blobs.
• [ ] Prompt injection of recalls is token-budgeted.
• [ ] TTL/ISM or sweeper for 3–14 day scratch horizon.
• [ ] Degrade path if OpenSearch errors (Redis-only).
• [ ] Embed model id pinned in SSM; dimension matches mapping.
• [ ] Complement — not replace — Redis exact scratchpads.

Semantic scratch memory makes multi-turn coding agents feel like they “remember last Tuesday’s failed auth refactor” — without pretending Redis GET is a vector database.

Prompt packing without poisoning the planner

Bad recall injection looks like dumping five paragraphs of prior failures into the system prompt. Good packing:

## Prior similar attempts (tenant-scoped, last 7d)
- [fail][typecheck] src/auth/middleware.ts — missing UserContext after rename (session s_0918)
- [fail][apply_patch] patch rejected — import cycle auth↔users (session s_0919)
- [ok][run_tests] fixed by introducing shared types package (session s_0920)
Use these as hints; verify against current tree before re-applying patches.

Rules:

1. Max 3–5 bullets; each ≤ 200 characters.
2. Prefer outcome=fail for repair tasks; mix for greenfield.
3. Never include secrets, tokens, or raw env dumps in summaries at index time.
4. Cite session_id so humans can drill into DynamoDB/S3 artifacts.
5. If cosine scores are weak (below your threshold), inject nothing — silence beats noise.

A/B this with eval traces in Athena — measure pass rate with vs without recall.

IAM sketch for indexer vs retriever

Split roles so a read-only planner path cannot delete the index:

{
  "Effect": "Allow",
  "Action": ["aoss:APIAccessAll"],
  "Resource": "arn:aws:aoss:us-east-1:123456789012:collection/colid"
}

Tighten with collection data-access policies: indexer Write/CreateIndex, retriever Read/DescribeCollectionItems only. Attach VPC endpoint policies so only the agent subnets reach the collection endpoint. Rotate nothing in env files — use task roles (KMS grants mindset).