9 views
Training-serving skew is the silent model killer: offline features computed in Spark, online features recomputed differently in the request path, AUC looks great in notebooks and awful in production. The choice between an online feature store and inline compute is really a choice about who owns parity, latency, and freshness.
⚡ TL;DR: Use an online store when many models share features or point-in-time correctness matters; use inline compute when features are cheap, local, and identical to training UDFs. Enforce shared code or generated definitions. Pair with RAG Evaluation on AWS mindset for continuous parity checks and SageMaker JumpStart vs Bedrock when hosting choices interact with feature SLAs.
Feature Stores vs Inline Compute: production guidance
| Criterion | Online feature store | Inline compute-on-read |
|---|---|---|
| Shared across models | Strong fit | Duplication risk |
| p99 budget | Prefetch / GetItem ~5–20 ms | CPU on request path |
| Freshness | Stream materialization lag | Instant from raw inputs |
| Point-in-time training | Native (event time joins) | Easy to get wrong |
| Ops cost | Store + pipelines | App complexity |
python
# features/choose.py
def choose(shared: bool, p99_ms: float, needs_pit: bool, cpu_us: float) -> str:
if needs_pit or shared:
return "online_store"
if cpu_us < 500 and p99_ms > 30:
return "inline" # ✅ cheap & local
if p99_ms < 15 and shared:
return "online_store_precompute"
return "hybrid_keys_inline_enrichment"
Kill skew with one definition
python
# features/definitions.py — shared by training + serving
from dataclasses import dataclass
@dataclass(frozen=True)
class AmountZScore:
name: str = "amount_zscore"
window_hours: int = 24
def compute(self, amount: float, mean: float, std: float) -> float:
return 0.0 if std == 0 else (amount - mean) / std
typescript
❌ Reimplementing “almost the same” SQL in the service — float edges and null handling will diverge.
// serving/features.ts — same formula, generated or shared lib
export function amountZScore(amount: number, mean: number, std: number) {
return std === 0 ? 0 : (amount - mean) / std; // ✅ identical to Python
}
Online store path on AWS
typescript
Materialize via Streams / Managed Streaming / SageMaker Feature Store ingestion; alarm on materialization lag vs training event-time assumptions.
// serving/fetch.ts
export async function fetchFeatures(entityId: string, names: string[]) {
const item = await ddb.get({
Key: { pk: `entity#${entityId}`, sk: "features" },
ProjectionExpression: names.join(","),
});
// ✅ Fail closed or fall back per model policy
if (!item.Item) throw new Error("features_missing");
return item.Item;
}
Parity tests in CI
python
def test_parity_amount_zscore():
samples = load_golden_batch("amount_zscore")
for s in samples:
offline = AmountZScore().compute(s.amount, s.mean, s.std)
online = inline_ts_equivalent(s.amount, s.mean, s.std)
assert abs(offline - online) < 1e-9
Closing checklist
✅ Dos – ✅ Choose store vs inline with latency + sharing + PIT criteria – ✅ Share feature definitions across train and serve – ✅ Alarm on materialization lag – ✅ Parity-test golden batches in CI – ✅ Document fallback when features are missing ❌ Don’ts – ❌ Don’t reimplement features twice by hand – ❌ Don’t ignore null/default mismatches – ❌ Don’t put heavy joins on the inline request path – ❌ Don’t train on future data the online path cannot see – ❌ Don’t treat feature store lag as “eventual” without model impact analysisRelated reading
- SageMaker JumpStart vs Bedrock: When Fine-Tunes Beat Prompting
- RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves
- CQRS Boundary Criteria
- DynamoDB Hot Key Mitigation
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
