Python OpenTelemetry Spans: Useful Traces Without Decorator Soup Everywhere

Decorating every service method with @tracer.start_as_current_span creates noise, drift, and merge conflicts—while still missing the spans that matter (DB drivers, HTTP clients, brokers). Prefer auto-instrumentation for frameworks, then add a few manual spans at business boundaries with stable names and attributes. Sampling keeps the bill and the UI usable.

⚡ TL;DR: Enable OTEL instrumentors for FastAPI/aiohttp/botocore/redis; set service.name and resource attrs once; add manual spans only at domain edges (checkout.authorize, inventory.reserve); use parent-based + ratio sampling. Align with Lambda ADOT vs Powertools: Tracing Tradeoffs on Node 20 Runtimes and Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents.

Auto-instrument first

# app/telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.aiohttp_client import AioHttpClientInstrumentor
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor

def setup(app, service: str = "checkout-api"):
    provider = TracerProvider(
        resource=Resource.create({"service.name": service, "deployment.environment": "prod"})
    )
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)
    FastAPIInstrumentor.instrument_app(app)
    AioHttpClientInstrumentor().instrument()
    BotocoreInstrumentor().instrument()

# ❌ Decorator soup — fragile and incomplete
@tracer.start_as_current_span("svc.a")
def a(): ...
@tracer.start_as_current_span("svc.b")
def b(): ...

Manual spans only at domain boundaries

Name spans after business operations, not functions. Attach stable attributes (tenant.id, order.id)—never raw PII.

# app/checkout.py
from opentelemetry import trace

tracer = trace.get_tracer("checkout")

async def authorize(order_id: str, tenant: str, charge):
    with tracer.start_as_current_span("checkout.authorize") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("tenant.id", tenant)
        result = await charge()
        span.set_attribute("payment.status", result.status)
        return result

Keep traces readable

For high RPS, parent-based sampling with a low default ratio and “always on” for error traces mirrors the Node guidance in OpenTelemetry Sampling for Node when that post is live—or apply the same X-Ray cost tradeoffs linked above.

from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio

provider = TracerProvider(
    resource=Resource.create({"service.name": "checkout-api"}),
    sampler=ParentBasedTraceIdRatio(0.05),
)

Propagation and workers

Ensure W3C traceparent crosses API Gateway → Lambda → SQS attributes → downstream workers. When you enqueue, inject context into message attributes; when you consume, extract before handling. Missing propagation is why “the trace dies at the queue.”

Closing checklist

✅ Dos
– ✅ Auto-instrument frameworks and AWS SDK
– ✅ Manual spans only for domain operations
– ✅ Stable span names + low-cardinality attributes
– ✅ Batch export with parent-based sampling
– ✅ Propagate context across queues and async tasks

❌ Don’ts
– ❌ Don’t decorate every function
– ❌ Don’t put PII or unbounded IDs in attributes
– ❌ Don’t use sync exporters on the request path
– ❌ Don’t sample at 100% in production forever
– ❌ Don’t invent proprietary trace headers when W3C works

Related reading

• Lambda ADOT vs Powertools: Tracing Tradeoffs on Node 20 Runtimes
• Lambda X-Ray Sampling: Cost Versus Debuggability During Incidents
• Lambda Telemetry API Extensions: Custom Samplers Under Five Milliseconds
• Lambda Log Buffering: Cut CloudWatch Ingestion Without Losing Correlation

Last updated on September 11, 2026