Fine-Tuning Titan: Internal SDK Autocomplete Without Public GitHub Noise

Foundation models know lodash better than your internal @acme/payments-sdk. Autocomplete then invents helpers that never existed. Fine-tuning (or a Titan adapter / continued pretrain on Bedrock custom fine-tuning flows) on curated internal call sites teaches the model your real APIs — if you avoid poisoning the set with secrets and deprecated junk.

⚡ TL;DR: Mine high-quality call sites from the monorepo, strip PII/secrets, format as completion examples, fine-tune a Titan (or approved Bedrock custom model) adapter, and evaluate next-token / exact-API accuracy on a frozen holdout. Prefer RAG+rules for rapidly changing APIs; fine-tune for stable internal SDKs. Illustrative bar: ≥ top-1 API-symbol accuracy 15+ points over base model on holdout.

When fine-tune beats RAG alone

Also compare JumpStart vs Bedrock prompting economics before committing training spend.

Curate call sites — quality over volume

# mine_calls.py — illustrative
import ast, json
from pathlib import Path

EXAMPLES = []

class Visitor(ast.NodeVisitor):
    def visit_Call(self, node: ast.Call):
        if isinstance(node.func, ast.Attribute):
            name = node.func.attr
            if name.startswith("_"):
                return
            # keep typed, non-test production calls only
        self.generic_visit(node)

for path in Path("packages").rglob("*.py"):
    if "test" in path.parts:  # exclude tests if they mock APIs unrealistically
        continue
    src = path.read_text(encoding="utf-8", errors="ignore")
    if "AKIA" in src or "BEGIN PRIVATE" in src:
        continue  # secrets — drop entire file
    # Build prefix/suffix completion pairs around SDK imports

Format examples as:

{"prompt": "from acme_payments import Client\nclient = Client()\ninvoice = client.", "completion": "create_invoice(\n    customer_id=customer_id,\n    amount_cents=amount,\n)\n"}

✅ Deduplicate near-identical prefixes.
❌ Train on generated code that already hallucinated APIs.

Scan training shards with gitleaks before upload — same hygiene as Claude Projects vs Cursor context.

Train and pin evaluation

# Illustrative Bedrock fine-tuning job sketch
aws bedrock create-model-customization-job \
  --job-name titan-acme-sdk-20260911 \
  --custom-model-name titan-acme-sdk-v3 \
  --role-arn arn:aws:iam::123456789012:role/BedrockFT \
  --base-model-identifier amazon.titan-text-express-v1 \
  --training-data-config s3Uri=s3://ml-prod/ft/acme-sdk/train.jsonl \
  --output-data-config s3Uri=s3://ml-prod/ft/acme-sdk/out/ \
  --hyper-parameters epochCount=2,batchSize=8,learningRate=0.00001

Holdout metrics that matter for autocomplete:

1. Exact API symbol match at cursor
2. Argument name F1 vs SDK signature
3. Compile rate of accepted suggestions in a shadow IDE study
4. Hallucinated symbol rate (should fall)

def exact_symbol_acc(preds, labels):
    return sum(p == y for p, y in zip(preds, labels)) / len(labels)

Serve without letting public GitHub noise back in

At inference, keep a negative constraint list of banned APIs and prefer retrieving internal SDK docs (RAG on OpenSearch vs pgvector). Fine-tune ≠ permission to skip Guardrails (Bedrock Agents guardrails).

Version the adapter (v3) and run canaries beside the base model before making it the IDE default. Budget training and inference under LLM cost controls.

Data contracts and refresh cadence

Internal SDKs evolve. Schedule a monthly mining job that:

1. Rebuilds train/holdout from main SHAs
2. Drops symbols marked @deprecated for > 30 days
3. Fails if secret scanners hit any shard
4. Trains only when new unique call sites exceed a threshold (e.g. 5% churn)

# illustrative gate
NEW=$(python3 tools/count_new_calls.py --since titan-acme-sdk-v3)
if [ "$NEW" -lt 500 ]; then
  echo "skip FT — not enough new signal ($NEW)"; exit 0
fi

Publish evaluation reports next to the model card. If hallucinated-symbol rate rises after an SDK major, roll back the IDE pointer to the previous adapter and lean on RAG until the next train. Keep autocomplete suggestions behind the same secret-redaction path as interactive chat.

Closing checklist

✅ Dos
– ✅ Mine production call sites; strip secrets and deprecated modules
– ✅ Freeze a holdout with exact-symbol metrics
– ✅ Pin base model + hyperparameters in the job name
– ✅ Canary the adapter against base on real IDE telemetry
– ✅ Combine with RAG for freshly renamed APIs

❌ Don’ts
– ❌ Don’t fine-tune on unredacted customer code dumps
– ❌ Don’t judge success only by training loss
– ❌ Don’t auto-roll adapters to all engineers on day one
– ❌ Don’t expect fine-tunes to replace least-privilege tool sandboxes
– ❌ Don’t mix multiple unrelated SDKs into one low-data adapter

Related reading

• RAG on AWS: OpenSearch vs Aurora pgvector for Codebase Chat
• Amazon Bedrock Agents: Tool Use, Memory, and Production Guardrails
• Bedrock Prompt Caching and Batch Inference: Cut Latency and Cost
• Claude Projects vs Cursor: Context Hygiene for Regulated Codebases (companion)

Last updated on September 11, 2026