Day 1: Tokens, Context Windows, and Why Models Forget Mid-Task

Naïve chatbots fail mid-task for boring reasons: they run out of tokens, they forget the middle of the prompt, and they treat the last user message as absolute truth. Day 1 is measurement day. If you cannot count tokens with the real tokenizer and prove where attention collapses on your prompts, every later architecture choice is superstition dressed as best practice.

⚡ TL;DR: Count tokens with the deployer’s tokenizer (not len(words)*1.3). Budget system + tools + retrieved chunks + history + user turn separately. Expect “lost in the middle.” Cap history with summarization or sliding windows — never hope the model remembers turn 40 unaided.

Tokens are the unit of truth

Models do not see characters. They see token IDs from a model-specific BPE or Unigram vocabulary. The same English sentence can be 18 tokens on one model and 27 on another. Code, JSON, and IAM ARNs inflate harder than prose: a compact-looking CloudFormation snippet can burn thousands of tokens because identifiers rarely merge into single pieces.

# ✅ Measure with the tokenizer you will actually deploy
import tiktoken  # or Anthropic/Bedrock tokenizer bindings for your model

enc = tiktoken.get_encoding("cl100k_base")

def budget(parts: dict[str, str], limit: int = 128_000) -> dict:
    counts = {k: len(enc.encode(v)) for k, v in parts.items()}
    counts["total"] = sum(counts.values())
    counts["headroom"] = limit - counts["total"]
    return counts

print(budget({
    "system": SYSTEM,
    "tools": TOOL_SCHEMAS_JSON,
    "rag": "\n\n".join(chunks),
    "history": history_text,
    "user": user_msg,
}))

❌ Estimating production cost and truncation risk with len(text)//4. That heuristic under-counts JSON keys, Unicode, and code identifiers and will surprise you at p95 billable tokens.

Log per-region counts on every request in staging for a week. You will discover that tool schemas or chat history — not the user question — dominate the bill.

Context windows are not free RAM

A 128k or 200k window is a marketing ceiling. Effective context is what you can fill while keeping latency, cost, and accuracy acceptable. Dumping an entire wiki into the prompt usually hurts answer quality because of recency bias and under-attention to the middle of long sequences.

Use an explicit packing policy for coding and ops assistants:

When headroom goes negative, drop lowest-ranked RAG chunks first, then older turns, then optionally compress scratch with a summarizer. Do not silently truncate the system policy.

Why models “forget” mid-task

Three mechanisms show up repeatedly in production traces:

1. Hard truncation — your client drops oldest messages or middle chunks when over budget.
2. Soft forgetting — the model still receives the tokens but under-attends content placed mid-context (“lost in the middle”).
3. Instruction collision — later user text overrides earlier system rules unless you restate hard constraints each turn.

# ✅ Probe lost-in-the-middle before you ship long RAG packs
def needle_test(ctx_passages: list[str], needle: str, ask: str) -> str:
    mid = len(ctx_passages) // 2
    packed = ctx_passages[:mid] + [needle] + ctx_passages[mid:]
    return call_model(system=SYSTEM, user="\n\n".join(packed) + "\n\n" + ask)

# Place a unique fact at ~50% depth; fail the design if recall < target

If the model cannot retrieve a unique operational fact placed at 40–60% depth in a realistic pack, your “big context” architecture is theater. Fix packing and retrieval before buying a larger window.

Architect for forgetfulness on purpose

Treat memory as unreliable and design around it:

• Re-pin critical constraints every turn: tenant ID, cite-or-refuse, “never invent ARNs.”
• Summarize closed work; keep raw tokens only for open tasks and the active diff.
• Prefer tools over memory for facts that must be exact (ticket IDs, IAM policy documents, current alarm state).
• Measure before model shopping — a disciplined 32k pack often beats a chaotic 200k dump on faithfulness.

For multi-hour agent sessions, store durable state outside the prompt (Day 13). The context window is a working set, not a database.

Closing checklist

• [ ] Tokenize with the real model tokenizer; log per-region counts in staging
• [ ] Separate budgets for system / tools / RAG / history / user
• [ ] Run a needle-in-haystack at mid-context on your corpus
• [ ] Cap chat history; summarize or drop — never infinite-append
• [ ] Restate hard constraints every turn for long sessions
• [ ] Alert when headroom is consistently < 10% of the window

Worked example: budgeting a coding-agent turn

Imagine a PR-review agent turn with: 1.8k system policy, 4.2k tool schemas, 9.5k retrieved CODEOWNERS + diff hunks, 3.1k summarized prior notes, and a 0.7k user ask. On a 128k model you have headroom — but TTFT and cost already hurt. On a 32k effective budget after latency caps, you must drop low-rank hunks and shrink schemas.

regions = {"system": 1800, "tools": 4200, "rag": 9500, "history": 3100, "user": 700}
limit = 24_000  # interactive class
order = ["rag", "history", "tools"]  # drop soft regions first; never drop system/user
total = sum(regions.values())
for k in order:
    while total > limit and regions[k] > 0:
        cut = min(500, regions[k], total - limit)
        regions[k] -= cut
        total -= cut
assert regions["system"] == 1800 and regions["user"] == 700

Production teams that skip this exercise discover truncation only when the model “forgets” the security policy mid-incident.

Failure modes to watch

• Silent middle drop in client libraries that trim “evenly.”
• Tokenizer mismatch between local tiktoken estimates and Bedrock billing tokens.
• History balloons from tool transcripts pasted verbatim each turn.
• False confidence after upgrading context window without re-running needle tests.

Field notes from production

Teams that only log total tokens miss that tool schemas alone can exceed the user question by 10×. Put a dashboard panel for tokens.tools vs tokens.user. When tools dominate, split schemas by agent role or lazy-load tool definitions per turn. Also: multilingual user text changes tokenizer behavior — re-run budgets for your top locales.

Implementation sketch

# Implementation sketch: middleware that rejects over-budget requests early
class TokenBudgetMiddleware:
    def __init__(self, limit: int, enc):
        self.limit, self.enc = limit, enc
    def check(self, parts: dict[str, str]):
        total = sum(len(self.enc.encode(v)) for v in parts.values())
        if total > self.limit:
            raise OverBudget(total=total, limit=self.limit)

Series navigation

Day 2 →

Last updated September 11, 2026