9 views
CloudWatch Logs anomaly detection is noisy the week you ship structured logging. The unfair advantage is teaching detectors what “normal deploy churn” looks like—suppression windows, baseline freezes, and focused metric filters—so novel ECONNRESET shapes still page while expected field additions do not.
⚡ TL;DR: Baseline on steady traffic; suppress or retrain across known deploys; alert on anomaly + error-rate composite; keep a Logs Insights saved query for novel message clusters. Pair with CI Failure Triage Bots and Lambda Log Buffering.
CloudWatch Logs Anomaly Detection: production guidance
text
Deploy → new fields / renamed codes → anomaly spike (benign)
Regression → new exception class / surge of rare code → anomaly + SLO burn (page)
| Technique | Effect |
|---|---|
| Suppression window around CodeDeploy/ECS events | Ignore expected shape change |
Metric filter on level=error rate |
Cheap primary signal |
| Anomaly on rare error codes only | High precision |
| Embed deploy marker in logs | Correlates false positives |
python
# ✅ Emit explicit deploy marker once per task start
import logging, os
logging.getLogger("boot").info(
"deploy_marker",
extra={"git_sha": os.environ["GIT_SHA"], "task_def": os.environ["TASK_DEFINITION"]},
)
Composite alarms beat raw anomaly
bash
# ✅ Alarm when: anomaly detector state ALARM
# AND error_rate > 1% for 5m
# AND NOT within 15m of a successful ECS deployment event
sql
❌ Paging on every anomaly without a human-tuned denylist — on-call will silence the detector permanently.
-- ✅ Logs Insights: novel messages after deploy (sample)
fields @timestamp, msg, error_code
| filter level = "error"
| stats count() as c by error_code, msg
| sort c desc
| limit 50
Retrain discipline
After intentional log schema changes, retrain/baseline for 24–48h of normal traffic. Document the change in the same PR as the schema bump. Keep canaries (Continuous Path Verification) as the source of truth for business path health.Closing checklist
✅ Dos – ✅ Use composite alarms (anomaly ∩ error-rate ∩ not-deploying) – ✅ Log deploy markers with git SHA – ✅ Maintain denylist for known noisy messages – ✅ Retrain after structured-log migrations – ✅ Review anomaly FP/FN weekly for two sprints after enablement ❌ Don’ts – ❌ Don’t alert solely on anomaly score – ❌ Don’t change log schemas without notifying detectors – ❌ Don’t sample away all errors before detection – ❌ Don’t let silenced detectors stay silencedRelated reading
- CI Failure Triage Bots: Separate Flaky Noise From Real Regressions
- Lambda Log Buffering: Cut CloudWatch Ingestion
- Continuous Path Verification: Minute Canaries
- Incident ChatOps AI: Draft Change Tickets
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
