CloudWatch Logs Anomaly Detection: Ignore Deploy Noise, Catch Novel Errors

CloudWatch Logs anomaly detection is noisy the week you ship structured logging. The unfair advantage is teaching detectors what “normal deploy churn” looks like—suppression windows, baseline freezes, and focused metric filters—so novel ECONNRESET shapes still page while expected field additions do not.

⚡ TL;DR: Baseline on steady traffic; suppress or retrain across known deploys; alert on anomaly + error-rate composite; keep a Logs Insights saved query for novel message clusters. Pair with CI Failure Triage Bots and Lambda Log Buffering.

Separate deploy noise from novelty

Deploy → new fields / renamed codes → anomaly spike (benign)
Regression → new exception class / surge of rare code → anomaly + SLO burn (page)

# ✅ Emit explicit deploy marker once per task start
import logging, os
logging.getLogger("boot").info(
    "deploy_marker",
    extra={"git_sha": os.environ["GIT_SHA"], "task_def": os.environ["TASK_DEFINITION"]},
)

Composite alarms beat raw anomaly

# ✅ Alarm when: anomaly detector state ALARM
# AND error_rate > 1% for 5m
# AND NOT within 15m of a successful ECS deployment event

-- ✅ Logs Insights: novel messages after deploy (sample)
fields @timestamp, msg, error_code
| filter level = "error"
| stats count() as c by error_code, msg
| sort c desc
| limit 50

❌ Paging on every anomaly without a human-tuned denylist — on-call will silence the detector permanently.

Retrain discipline

After intentional log schema changes, retrain/baseline for 24–48h of normal traffic. Document the change in the same PR as the schema bump. Keep canaries (Continuous Path Verification) as the source of truth for business path health.

Closing checklist

✅ Dos
– ✅ Use composite alarms (anomaly ∩ error-rate ∩ not-deploying)
– ✅ Log deploy markers with git SHA
– ✅ Maintain denylist for known noisy messages
– ✅ Retrain after structured-log migrations
– ✅ Review anomaly FP/FN weekly for two sprints after enablement

❌ Don’ts
– ❌ Don’t alert solely on anomaly score
– ❌ Don’t change log schemas without notifying detectors
– ❌ Don’t sample away all errors before detection
– ❌ Don’t let silenced detectors stay silenced

Related reading

• CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
• Lambda Log Buffering: Cut CloudWatch Ingestion
• Continuous Path Verification: Minute Canaries
• Incident ChatOps AI: Draft Change Tickets

Last updated on September 11, 2026