Private Model Hosting: When Bedrock Alone Cannot Meet Constraints

Bedrock is the right default until it is not. Custom weights, hard residency, sub-80ms p95 autocomplete, or air-gapped training artifacts force private hosting. The unfair advantage is a blunt decision tree — SageMaker real-time / async / JumpStart vs Bedrock — so platform teams stop debating vibes and ship the constraint that actually binds.

⚡ TL;DR: Stay on Bedrock unless you need custom weights, residency outside Bedrock regions, or latency/throughput Bedrock cannot meet. Prefer SageMaker real-time + PrivateLink; keep Guardrails/IAM wrappers either way. Revisit quarterly. Pair with SageMaker JumpStart vs Bedrock and AI Coding in VPC.

Decision tree (bind on the first true)

1. Need weights you train/own that Bedrock Custom Model Import cannot serve?
   → SageMaker (or self-managed GPU) with your artifact + CMK
2. Data residency / sovereign region without Bedrock model access?
   → Private hosting in that region; deny cross-region invokes via SCP
3. Hard p95 < Bedrock floor for your token shape (measure, don't guess)?
   → SageMaker real-time + provisioned concurrency / Triton
4. Otherwise
   → Bedrock (+ Prompt Management, Guardrails, inference profiles)

# policy/hosting_choice.py
from dataclasses import dataclass

@dataclass
class Constraints:
    custom_weights: bool
    residency_ok_on_bedrock: bool
    p95_ms_budget: int
    bedrock_p95_ms: int  # measured on canary

def choose(c: Constraints) -> str:
    if c.custom_weights:
        return "sagemaker_realtime"
    if not c.residency_ok_on_bedrock:
        return "sagemaker_or_onprem"
    if c.p95_ms_budget < c.bedrock_p95_ms:
        return "sagemaker_realtime"
    return "bedrock"  # ✅ default

❌ “We host everything privately for control” with no residency or latency proof — you buy ops tax for free.

SageMaker real-time pattern that does not leak

# infra/sagemaker-endpoint.yaml (conceptual)
EndpointConfig:
  ProductionVariants:
    - ModelName: coding-assistant-v3
      InitialInstanceCount: 2
      InstanceType: ml.g5.xlarge
VpcConfig:
  Subnets: [subnet-private-a, subnet-private-b]
  SecurityGroupIds: [sg-model-only]
# ✅ No public internet; PrivateLink to caller VPC
# ✅ Invoke via IAM authz, not long-lived API keys

Wrap invokes with the same secret hygiene and audit as Bedrock — see Cross-Account Bedrock Access for attribution tags and SCPs.

Cost and ops reality check

If idle GPU burn exceeds Bedrock spend at your QPS, you chose wrong — measure with LLM Cost Controls.

Closing checklist

✅ Dos
– ✅ Bind on first true constraint in the tree
– ✅ Measure Bedrock p95 on your prompts before migrating
– ✅ PrivateLink + IAM; no public model endpoints
– ✅ Keep audit/Guardrails wrappers on private hosts
– ✅ Re-evaluate when Bedrock regions/models expand

❌ Don’ts
– ❌ Don’t private-host for fashion
– ❌ Don’t skip SCPs that block accidental public exposes
– ❌ Don’t forget patch SLAs on GPU AMIs
– ❌ Don’t mix tenant traffic without isolation
– ❌ Don’t abandon Bedrock Prompt Management habits after moving

Related reading

• SageMaker JumpStart vs Bedrock: When Fine-Tunes Beat Prompting
• AI Coding in VPC: Private Bedrock Endpoints and Secret Hygiene
• Cross-Account Bedrock Access: Platform Teams Without Shared Keys
• LLM Cost Controls: Token Budgets Per PR and Per Engineer

Last updated on September 11, 2026