10 views
Decorating every service method with @tracer.start_as_current_span creates noise, drift, and merge conflicts—while still missing the spans that matter (DB drivers, HTTP clients, brokers). Prefer auto-instrumentation for frameworks, then add a few manual spans at business boundaries with stable names and attributes. Sampling keeps the bill and the UI usable.
⚡ TL;DR: Enable OTEL instrumentors for FastAPI/aiohttp/botocore/redis; setservice.nameand resource attrs once; add manual spans only at domain edges (checkout.authorize,inventory.reserve); use parent-based + ratio sampling. Align with Lambda ADOT vs Powertools: Tracing Tradeoffs on Node 20 Runtimes and Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents.
Python OpenTelemetry Spans: production guidance
python
# app/telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.aiohttp_client import AioHttpClientInstrumentor
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
def setup(app, service: str = "checkout-api"):
provider = TracerProvider(
resource=Resource.create({"service.name": service, "deployment.environment": "prod"})
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
FastAPIInstrumentor.instrument_app(app)
AioHttpClientInstrumentor().instrument()
BotocoreInstrumentor().instrument()
python
# ❌ Decorator soup — fragile and incomplete
@tracer.start_as_current_span("svc.a")
def a(): ...
@tracer.start_as_current_span("svc.b")
def b(): ...
Manual spans only at domain boundaries
Name spans after business operations, not functions. Attach stable attributes (tenant.id, order.id)—never raw PII.
python
# app/checkout.py
from opentelemetry import trace
tracer = trace.get_tracer("checkout")
async def authorize(order_id: str, tenant: str, charge):
with tracer.start_as_current_span("checkout.authorize") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("tenant.id", tenant)
result = await charge()
span.set_attribute("payment.status", result.status)
return result
Keep traces readable
| Do | Don’t |
|---|---|
| One span per external call (auto) | Span per tiny helper |
checkout.authorize |
helpers.py:do_stuff |
Record exceptions with span.record_exception |
Swallow and lose status |
| Batch export | Sync export on request path |
python
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio
provider = TracerProvider(
resource=Resource.create({"service.name": "checkout-api"}),
sampler=ParentBasedTraceIdRatio(0.05),
)
Propagation and workers
Ensure W3Ctraceparent crosses API Gateway → Lambda → SQS attributes → downstream workers. When you enqueue, inject context into message attributes; when you consume, extract before handling. Missing propagation is why “the trace dies at the queue.”
Closing checklist
✅ Dos – ✅ Auto-instrument frameworks and AWS SDK – ✅ Manual spans only for domain operations – ✅ Stable span names + low-cardinality attributes – ✅ Batch export with parent-based sampling – ✅ Propagate context across queues and async tasks ❌ Don’ts – ❌ Don’t decorate every function – ❌ Don’t put PII or unbounded IDs in attributes – ❌ Don’t use sync exporters on the request path – ❌ Don’t sample at 100% in production forever – ❌ Don’t invent proprietary trace headers when W3C worksRelated reading
- Lambda ADOT vs Powertools: Tracing Tradeoffs on Node 20 Runtimes
- Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents
- Lambda Telemetry API Extensions: Custom Samplers Under Five Milliseconds
- Lambda Log Buffering: Cut CloudWatch Ingestion Without Losing Correlation
Watch: 100 Days of AI on YouTube
Short videos from CheatCoders, one AI engineering topic per day.
Most viewed
- Python Decorators Explained: From Simple Wrappers to Production Patterns
- Claude Code vs Cursor vs GitHub Copilot: Choose by Workflow, and Set Up One Repo That Works With All Three
- AI Agent Frameworks in 2025: LangGraph vs CrewAI vs AutoGen vs Raw API
- Python String Methods: Every str Method With Real Production Examples
- Css Animation Performance: Performance-Optimized Motion for Modern UIs
Newly added
- AWS Systems Manager Parameter Store: Hierarchical Runtime Config for Coding Agents
- Amazon Cognito Identity Pools: Federate Coding-Agent Tool Callers Without Long-Lived Keys
- Amazon Athena for Coding Agents: Query Tool Audit Logs and Eval Traces on S3 Without a Warehouse
- AWS Batch for Coding Agents: Parallel Overnight PR Reviews and Long-Running Jobs Without Lambda Timeouts
- Amazon Bedrock Prompt Management: Version and A/B Test Coding-Agent System Prompts
Deep-dive PDF
Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.
Already subscribed? or open the subscribe page.
Discover more from CheatCoders
Subscribe to get the latest posts sent to your email.
