SageMaker JumpStart vs Bedrock: When Fine-Tuning Beats Prompting and RAG for Coding Assistants

Someone on your team says the assistant “doesn’t know our SDK” and proposes fine-tuning a model. Sometimes that’s the right call. More often the real problem is that retrieval never surfaces the right file, or the prompt never states the house rules. Fine-tuning then costs weeks of data work and still answers with last quarter’s API names.

This guide is the decision gate to run before anyone opens a training ticket. It explains what supervised fine-tuning can and can’t teach, compares Amazon Bedrock model customization with SageMaker JumpStart, and walks through a complete Bedrock fine-tune for a coding use case: mining training data from merged pull requests, submitting the job, deploying it on demand and evaluating it against a frozen baseline. It merges two earlier bootcamp lessons, on the fine-tuning decision and on fine-tuning for internal SDK autocomplete, into one guide.

⚡ TL;DR

• Fix prompts, retrieval and tools first. AWS’s Nova documentation states it plainly: supervised fine-tuning teaches behaviors, response formats and reasoning patterns, not new facts. Facts about your SDK belong in RAG.
• Fine-tune when the failure is behavior: a house style the model won’t hold, a fixed output format, a terse dialect, or a smaller model that must match a larger one on a narrow task.
• Bedrock customization is managed: JSONL in S3, a job, a custom model you call by ARN. SageMaker JumpStart gives you more model and training choice, but you own the training jobs and endpoints.
• Require a frozen eval set and a prompt-plus-RAG baseline before training, and a rollback path to prompt-only after.

Who this is for: platform and ML engineers deciding whether to customize a model for an internal coding assistant. Code is Python with boto3. Earlier versions of this guide used Titan Text as the default fine-tuning target; see the corrections below.

What fine-tuning changes, and what it doesn’t

A foundation model reads three things at inference time: its weights, the prompt and anything you retrieve into the prompt. Prompting changes instructions. Retrieval changes the facts in context. Fine-tuning changes the weights, so it changes default behavior on every request.

That makes the decision mostly about the kind of failure you’re seeing:

The common mistake is reading “doesn’t know our SDK” as a weights problem. If the right file was never retrieved, no amount of training on last month’s call sites fixes next month’s renames. Measure retrieval hit rate on your eval questions first.

Bedrock customization vs SageMaker JumpStart

The Bedrock fine-tuning table is Region-specific: at the time of writing, the Nova models are listed for us-east-1, and Claude 3 Haiku and the Llama 3.x models for us-west-2. Check the table before you plan, because you train and serve the custom model in that Region.

On-demand inference for a custom model works through a custom model deployment: you deploy the model and use the deployment’s ARN as the modelId. The documentation currently lists Nova Micro, Lite, Pro and Nova 2 Lite (us-east-1) and Llama 3.3 70B Instruct (us-west-2) as supported for on-demand deployment, for models customized on or after July 16, 2025. Other custom models need Provisioned Throughput, which is billed by the hour whether or not you send traffic. That cost difference often decides the question for internal tools with bursty traffic.

Training is charged by tokens processed (training tokens times epochs), plus monthly model storage. Add the labeling and review hours to your estimate; they’re usually the largest cost.

Step 1: Build a frozen eval set and a baseline

Before any training data exists, collect a few hundred real tasks from your tickets and pull requests, each with an accepted answer: the merged diff, the reviewer-approved snippet or the expected tool call. Freeze it, version it and never train on it. Run your current best setup (prompt, few-shot examples, retrieval) against it and record the score. Write the threshold that would justify training into the decision record now. Choosing the bar after you see the results is how teams talk themselves into a fine-tune.

Step 2: Mine training examples from merged PRs

For a coding assistant, the best training data is work your reviewers already approved: merged diffs, accepted review suggestions and corrected agent patches. Each example should demonstrate the behavior you want, using the same system prompt you’ll use in production. AWS’s Nova data guidance recommends exactly this: real user queries, gold-standard responses, one skill per sample, and quality over quantity.

For conversational models, Bedrock uses the bedrock-conversation-2024 format: one JSON object per line with schemaVersion, an optional system list and alternating user and assistant messages that start with the user and end with the assistant.

import hashlib
import json
import re
from pathlib import Path

SYSTEM = [{"text": "You write Python for the acme platform. Follow the house conventions: "
                   "typed signatures, structured logging via log.info(event=...), "
                   "and AcmeError subclasses for failures. Return only the code."}]

SECRET_PATTERNS = [
    re.compile(r"AKIA[0-9A-Z]{16}"),                          # AWS access key ID
    re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    re.compile(r"(?i)(api[_-]?key|secret|password|token)\s*[:=]\s*['\"][^'\"]{8,}"),
]
# Nova training jobs fail if prompts contain these reserved keywords.
NOVA_RESERVED = ("User:", "Bot:", "Assistant:", "System:", "<image>", "<video>", "[EOS]")


def clean(text: str) -> bool:
    return not any(p.search(text) for p in SECRET_PATTERNS)


def to_record(task: str, accepted_code: str) -> dict:
    return {
        "schemaVersion": "bedrock-conversation-2024",
        "system": SYSTEM,
        "messages": [
            {"role": "user", "content": [{"text": task}]},
            {"role": "assistant", "content": [{"text": accepted_code}]},
        ],
    }


def build(prs: list, eval_ids: set, out: Path) -> dict:
    """prs: [{'id': ..., 'task': PR description, 'code': final merged code}]"""
    seen, stats = set(), {"kept": 0, "secret": 0, "dup": 0, "eval_leak": 0, "reserved": 0}
    with out.open("w") as f:
        for pr in prs:
            if pr["id"] in eval_ids:
                stats["eval_leak"] += 1          # never train on the frozen eval set
                continue
            if not (clean(pr["task"]) and clean(pr["code"])):
                stats["secret"] += 1             # drop the whole example, don't redact
                continue
            if any(k in pr["task"] for k in NOVA_RESERVED):
                stats["reserved"] += 1
                continue
            key = hashlib.sha256(re.sub(r"\s+", " ", pr["code"]).encode()).hexdigest()
            if key in seen:
                stats["dup"] += 1
                continue
            seen.add(key)
            f.write(json.dumps(to_record(pr["task"], pr["code"])) + "\n")
            stats["kept"] += 1
    return stats

The regexes are a last line of defense, not a scanner. Run a real secret scanner such as gitleaks over the source data too, and drop examples from repositories you’re not licensed to train on. Exclude code that was generated by an assistant and merged without review: training on it teaches the model to repeat its own mistakes.

The Nova documentation lists a minimum of 8 and a maximum of 20,000 samples for Nova Micro, Lite and Pro, with a 32k context length, and recommends at least 100 samples per task you want the model to learn. Treat those as limits, not targets.

Step 3: Submit the customization job

A fine-tuning job needs a service role that can read the training data and write output in S3, a base model, and hyperparameters. For Nova Micro, Lite and Pro there are three: epochCount (1–5, default 2), learningRate (1e-6 to 1e-4, default 1e-5) and learningRateWarmupSteps (0–100, default 10). AWS recommends starting with the defaults. Hyperparameter values are passed as strings.

import time

import boto3

bedrock = boto3.client("bedrock", region_name="us-east-1")   # the Region listed for your base model

job = bedrock.create_model_customization_job(
    jobName="acme-style-sft-2026-10-10",
    customModelName="acme-style-nova-lite-v1",
    roleArn="arn:aws:iam::111122223333:role/BedrockCustomizationRole",
    baseModelIdentifier="amazon.nova-lite-v1:0:300k",          # from the fine-tuning table
    customizationType="FINE_TUNING",
    trainingDataConfig={"s3Uri": "s3://acme-ml/sft/train.jsonl"},
    validationDataConfig={"validators": [{"s3Uri": "s3://acme-ml/sft/validation.jsonl"}]},
    outputDataConfig={"s3Uri": "s3://acme-ml/sft/output/"},
    hyperParameters={"epochCount": "2", "learningRate": "0.00001", "learningRateWarmupSteps": "10"},
    clientRequestToken="acme-style-sft-2026-10-10",          # makes a retried submit idempotent
    customModelTags=[{"key": "team", "value": "dev-platform"}],
)

while True:
    status = bedrock.get_model_customization_job(jobIdentifier=job["jobArn"])
    if status["status"] in ("Completed", "Failed", "Stopped"):
        break
    time.sleep(300)                                          # jobs can take hours

if status["status"] != "Completed":
    raise SystemExit(f"customization ended {status['status']}: {status.get('failureMessage')}")

model_arn = status["outputModelArn"]
deployment = bedrock.create_custom_model_deployment(
    modelDeploymentName="acme-style-nova-lite-v1",
    modelArn=model_arn,
)
print("use as modelId once Active:", deployment["customModelDeploymentArn"])

Bedrock writes training and validation loss per epoch to the output location. Falling training loss with rising validation loss is the usual sign of overfitting: fewer epochs or more varied data, not a higher learning rate.

Step 4: Evaluate blind against the baseline

Run the frozen eval set through both the baseline (prompt plus RAG on the base model) and the custom model with the same system prompt, then score the outputs without telling reviewers which is which. Automated checks come first: does the code parse, type-check and pass the task’s tests? Human review comes second, for conventions and readability.

def decide(baseline: dict, candidate: dict, min_lift: float, max_regression: float) -> str:
    """Scores are pass rates on the frozen eval set, per category.
    min_lift and max_regression come from the decision record written before training."""
    lift = candidate["overall"] - baseline["overall"]
    regressions = {k: baseline[k] - candidate[k] for k in baseline
                   if k != "overall" and baseline[k] - candidate[k] > max_regression}
    if regressions:
        return f"NO-GO: regressed {regressions}"
    if lift < min_lift:
        return f"NO-GO: lift {lift:.3f} below agreed {min_lift}"
    return f"GO: lift {lift:.3f}; ship behind a flag with prompt-only rollback"

Check per-category regressions, not just the average. A fine-tune that improves style can quietly get worse at things you didn’t train on, such as explaining code or refusing unsafe requests. Keep guardrails on the custom model exactly as on the base model.

Worked example (illustrative): house logging conventions

1. The assistant keeps writing print() and bare except despite a system prompt that says otherwise. Retrieval works: the right modules are in context.
2. The team freezes 300 tasks from recent PRs. With few-shot examples the baseline follows conventions in most but not all cases, and the decision record sets the bar for a fine-tune.
3. They mine 2,000 merged changes from reviewed PRs, drop examples with secret-like strings, duplicates and eval overlap, and train Nova Lite with default hyperparameters in us-east-1.
4. Blind review shows the custom model follows conventions more consistently without regressions in other categories. It ships behind a flag, deployed on demand, with the base model plus prompt as rollback.
5. SDK method names still come from retrieval. When the SDK renames a method, the index updates and the fine-tune doesn’t need retraining.

Troubleshooting

Production checklist

• ☐ Decision record with the failure type, baseline score and go/no-go threshold, written before training
• ☐ Frozen, versioned eval set excluded from training data
• ☐ Training data from reviewed work only, with secret scanning, license review and deduplication
• ☐ Same system prompt in training and production
• ☐ Base model and Region checked against the fine-tuning and on-demand tables
• ☐ Cost model including labeling hours, training tokens times epochs, storage and inference (on demand vs Provisioned Throughput)
• ☐ Guardrails and invocation logging configured the same as for the base model
• ☐ Feature flag and prompt-only rollback; an owner for drift monitoring and retraining

FAQ

How many examples do I need?

For Nova Micro, Lite and Pro the documented range is 8 to 20,000 samples, and AWS recommends at least 100 per task. A few hundred consistent, reviewed examples typically beat thousands of noisy ones.

Can fine-tuning teach the model our internal APIs?

It can teach the shape of code that uses them, but AWS’s guidance is that supervised fine-tuning doesn’t teach new facts. Use retrieval for API names and signatures, and verify output with your compiler or type checker.

Can I still fine-tune Titan Text?

Titan Text models aren’t in the current Bedrock fine-tuning table of supported models, although some data-format pages still mention them. Plan new work on a model in the table, such as Amazon Nova or Llama, and verify before you start.

When should I pick SageMaker JumpStart?

When you need a model or training method Bedrock doesn’t offer, or full control of training and serving. You take on the training jobs, endpoints, scaling and patching. Custom Model Import can bring supported weights back into Bedrock for on-demand inference.

What about distillation or reinforcement fine-tuning?

Bedrock also offers distillation (a teacher model generates responses to fine-tune a smaller student) and reinforcement fine-tuning with reward functions. The same decision gate applies: frozen eval, baseline first.

Official documentation

• Customize your model to improve its performance
• Customize a model with fine-tuning: supported models and Regions
• Prepare data for fine-tuning your models
• Custom model hyperparameters
• Submit a model customization job
• Deploy a custom model for on-demand inference
• Custom Model Import
• SageMaker JumpStart: fine-tune a model

Related on CheatCoders: Bedrock Knowledge Bases for monorepos · Converse tool use and strict tool contracts · Token budgets per PR

Last updated on October 10, 2026