Feature Stores vs Inline Compute: Real-Time Decisions Without Training Skew

Training-serving skew is the silent model killer: offline features computed in Spark, online features recomputed differently in the request path, AUC looks great in notebooks and awful in production. The choice between an online feature store and inline compute is really a choice about who owns parity, latency, and freshness.

⚡ TL;DR: Use an online store when many models share features or point-in-time correctness matters; use inline compute when features are cheap, local, and identical to training UDFs. Enforce shared code or generated definitions. Pair with RAG Evaluation on AWS mindset for continuous parity checks and SageMaker JumpStart vs Bedrock when hosting choices interact with feature SLAs.

Decision matrix

# features/choose.py
def choose(shared: bool, p99_ms: float, needs_pit: bool, cpu_us: float) -> str:
    if needs_pit or shared:
        return "online_store"
    if cpu_us < 500 and p99_ms > 30:
        return "inline"  # ✅ cheap & local
    if p99_ms < 15 and shared:
        return "online_store_precompute"
    return "hybrid_keys_inline_enrichment"

Kill skew with one definition

# features/definitions.py  — shared by training + serving
from dataclasses import dataclass

@dataclass(frozen=True)
class AmountZScore:
    name: str = "amount_zscore"
    window_hours: int = 24

    def compute(self, amount: float, mean: float, std: float) -> float:
        return 0.0 if std == 0 else (amount - mean) / std

// serving/features.ts — same formula, generated or shared lib
export function amountZScore(amount: number, mean: number, std: number) {
  return std === 0 ? 0 : (amount - mean) / std; // ✅ identical to Python
}

❌ Reimplementing “almost the same” SQL in the service — float edges and null handling will diverge.

Online store path on AWS

// serving/fetch.ts
export async function fetchFeatures(entityId: string, names: string[]) {
  const item = await ddb.get({
    Key: { pk: `entity#${entityId}`, sk: "features" },
    ProjectionExpression: names.join(","),
  });
  // ✅ Fail closed or fall back per model policy
  if (!item.Item) throw new Error("features_missing");
  return item.Item;
}

Materialize via Streams / Managed Streaming / SageMaker Feature Store ingestion; alarm on materialization lag vs training event-time assumptions.

Parity tests in CI

def test_parity_amount_zscore():
    samples = load_golden_batch("amount_zscore")
    for s in samples:
        offline = AmountZScore().compute(s.amount, s.mean, s.std)
        online = inline_ts_equivalent(s.amount, s.mean, s.std)
        assert abs(offline - online) < 1e-9

Closing checklist

✅ Dos
– ✅ Choose store vs inline with latency + sharing + PIT criteria
– ✅ Share feature definitions across train and serve
– ✅ Alarm on materialization lag
– ✅ Parity-test golden batches in CI
– ✅ Document fallback when features are missing

❌ Don’ts
– ❌ Don’t reimplement features twice by hand
– ❌ Don’t ignore null/default mismatches
– ❌ Don’t put heavy joins on the inline request path
– ❌ Don’t train on future data the online path cannot see
– ❌ Don’t treat feature store lag as “eventual” without model impact analysis

Related reading

• SageMaker JumpStart vs Bedrock: When Fine-Tunes Beat Prompting
• RAG Evaluation on AWS: Hit Rate, Faithfulness, and Cost Curves
• CQRS Boundary Criteria
• DynamoDB Hot Key Mitigation

Last updated on September 11, 2026