Serverless API on AWS: API Gateway, Lambda, DynamoDB and SQS, From First Endpoint to Production

16 views

A serverless API on AWS usually starts as API Gateway, a Lambda function and a DynamoDB table, and that’s a good start. What breaks it in production isn’t scale in the abstract. It’s retries that create duplicate orders, a slow downstream call that holds a request open until the gateway times out, one client that uses up everyone’s throughput, and a worker that fails halfway through a batch. This guide builds the API with those problems handled, using documented limits rather than “hyper-scale” claims, and code that was tested.

⚡ TL;DR

Choose a REST API if you need API keys and per-client throttling, request validation, AWS WAF or private endpoints; choose an HTTP API if you don’t, because it’s offered at a lower price.

Know the limits: 10,000 requests per second per account and Region across API Gateway by default (with a 5,000-request burst bucket), a 29-second default integration timeout on REST APIs, and Lambda’s 1,000 default concurrency per Region.

Make every write idempotent: claim an Idempotency-Key with a conditional write, replay the stored response on retry, and reject reuse with a different body.

Keep the request path short: accept, store, queue, return 202. Do slow work in an SQS worker that dedupes and reports partial batch failures.

Figure 1. The request path stays short: API Gateway throttles and WAF filters, Lambda claims the idempotency key, writes the order and queues the job, and returns 202. An SQS worker does the slow part, with a dead-letter queue for jobs that keep failing. Diagram by CheatCoders.

REST API or HTTP API?

API Gateway offers two products for this. The AWS docs put it plainly: REST APIs support more features, HTTP APIs have minimal features so they can be offered at a lower price.

Need	REST API	HTTP API
API keys, per-client rate limiting and usage throttling (usage plans)	Yes	No
Request validation	Yes	No
AWS WAF	Yes	No
Private endpoints, edge-optimized endpoints	Yes	No (regional only)
Response caching, canary deployments, X-Ray tracing	Yes	No
Native JWT authorizer	No (use a Lambda authorizer or Cognito)	Yes
Automatic deployments	No	Yes

For a public API with paying clients, the REST API features (per-client throttling, WAF, validation) usually decide it. An earlier version of this post paired an HTTP API with usage plans and WAF; those features aren’t available on HTTP APIs.

The limits you’ll actually hit

Account throttle. By default, API Gateway allows 10,000 requests per second per account per Region, shared across HTTP, REST and WebSocket APIs, with a token-bucket burst of up to 5,000 requests. Some newer Regions start at 2,500 and 1,250. It can be increased.
Integration timeout. REST API integrations time out between 50 milliseconds and 29 seconds by default; for Regional and private APIs you can raise it, possibly at the cost of a lower Region-level throttle quota. HTTP APIs have a maximum of 30 seconds. Either way, anything slow belongs in a queue.
Lambda concurrency. Accounts start with 1,000 concurrent executions per Region across all functions. Concurrency is requests per second times average duration: 5,000 requests per second at 200 ms needs 1,000.
Lambda scaling rate. Each function can add up to 1,000 execution environments every 10 seconds. Requests beyond what it can serve are throttled with a 429.

Two consequences: a slow handler costs concurrency, so a 2-second call to a downstream service at 500 requests per second already needs the whole default quota; and throttling at the stage (below) is better than letting Lambda throttle, because it fails fast with a clear error.

Step 1: An idempotent write path

Clients, SDKs and gateways retry. Without idempotency, a retried POST /orders creates a second order. The fix is an Idempotency-Key header the client generates per logical request, and a handler that:

Claims the key with a conditional write (status IN_PROGRESS, a short lock, the body’s hash).
On conflict, compares hashes: a different body with the same key is a client bug (422); an in-progress claim gets a 409 so the client retries shortly; a completed one gets the stored response, byte for byte.
Derives the order ID from the tenant and key, so a retry after a crash writes the same order, not a new one.
Commits the order and the stored response in one DynamoDB transaction.

python

"""POST /orders behind API Gateway (REST, Lambda proxy). Idempotent by the Idempotency-Key header."""
import hashlib
import json
import os
import time
import uuid

import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
sqs = boto3.client("sqs")
TABLE, QUEUE_URL = os.environ["TABLE_NAME"], os.environ["QUEUE_URL"]
LOCK_SECONDS, KEEP_SECONDS = 30, 24 * 3600


def respond(code: int, body: dict) -> dict:
    return {"statusCode": code, "headers": {"Content-Type": "application/json"}, "body": json.dumps(body)}


def handler(event, context):
    tenant = event["requestContext"]["authorizer"]["tenant_id"]          # set by your authorizer, never by the client
    key = (event.get("headers") or {}).get("Idempotency-Key")
    if not key or len(key) > 128:
        return respond(400, {"error": "Idempotency-Key header (1-128 chars) is required"})
    body = event.get("body") or "{}"
    digest = hashlib.sha256(body.encode()).hexdigest()
    pk, now = {"S": f"TENANT#{tenant}"}, int(time.time())
    try:   # claim the key; a stale IN_PROGRESS claim (crashed invocation) can be taken over
        ddb.put_item(TableName=TABLE, Item={
            "pk": pk, "sk": {"S": f"IDEMP#{key}"}, "status": {"S": "IN_PROGRESS"}, "hash": {"S": digest},
            "lock_until": {"N": str(now + LOCK_SECONDS)}, "expires_at": {"N": str(now + KEEP_SECONDS)}},
            ConditionExpression="attribute_not_exists(sk) OR (#s = :p AND lock_until < :now)",
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={":p": {"S": "IN_PROGRESS"}, ":now": {"N": str(now)}})
    except ClientError as e:
        if e.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise
        rec = ddb.get_item(TableName=TABLE, Key={"pk": pk, "sk": {"S": f"IDEMP#{key}"}}, ConsistentRead=True)["Item"]
        if rec["hash"]["S"] != digest:
            return respond(422, {"error": "Idempotency-Key was already used with a different body"})
        if rec["status"]["S"] == "IN_PROGRESS":
            return respond(409, {"error": "a request with this key is still in progress; retry shortly"})
        return json.loads(rec["response"]["S"])                         # replay the stored response exactly

    order = json.loads(body)
    # Deterministic ID: if this invocation crashes and a retry takes over, it writes the same order, not a second one.
    order_id = str(uuid.uuid5(uuid.NAMESPACE_URL, f"{tenant}/{key}"))
    # Queue first. A crash after this line means the retry queues the same order_id again, so the consumer
    # must dedupe on order_id (standard SQS delivers at least once anyway).
    sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=json.dumps({"tenant": tenant, "order_id": order_id}))
    resp = respond(202, {"order_id": order_id, "status": "accepted"})
    ddb.transact_write_items(TransactItems=[           # the order and the stored response commit together
        {"Put": {"TableName": TABLE, "Item": {"pk": pk, "sk": {"S": f"ORDER#{order_id}"},
                                               "sku": {"S": order["sku"]}, "qty": {"N": str(int(order["qty"]))}}}},
        {"Update": {"TableName": TABLE, "Key": {"pk": pk, "sk": {"S": f"IDEMP#{key}"}},
                    "UpdateExpression": "SET #s = :c, #r = :r",
                    "ExpressionAttributeNames": {"#s": "status", "#r": "response"},
                    "ExpressionAttributeValues": {":c": {"S": "COMPLETE"}, ":r": {"S": json.dumps(resp)}}}}])
    return resp

The tenant comes from the authorizer context, never from the request body. The test runs the handler against moto’s DynamoDB and SQS, including a crash between queuing and committing:

python

import json, os, time
os.environ.update(AWS_DEFAULT_REGION="us-east-1", AWS_ACCESS_KEY_ID="x", AWS_SECRET_ACCESS_KEY="x")
from moto import mock_aws
import boto3

with mock_aws():
    ddb, sqs = boto3.client("dynamodb"), boto3.client("sqs")
    ddb.create_table(TableName="api", BillingMode="PAY_PER_REQUEST",
                     KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}],
                     AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}, {"AttributeName": "sk", "AttributeType": "S"}])
    q = sqs.create_queue(QueueName="work")["QueueUrl"]
    os.environ.update(TABLE_NAME="api", QUEUE_URL=q)
    import handler

    def call(key, body):
        ev = {"requestContext": {"authorizer": {"tenant_id": "t-42"}}, "headers": {"Idempotency-Key": key}, "body": json.dumps(body)}
        r = handler.handler(ev, None)
        return r["statusCode"], json.loads(r["body"])

    first = call("k-1", {"sku": "A-1", "qty": 2})
    print("first:   ", first)
    print("retry:   ", call("k-1", {"sku": "A-1", "qty": 2}), "same order:", call("k-1", {"sku": "A-1", "qty": 2})[1] == first[1])
    print("misuse:  ", call("k-1", {"sku": "A-1", "qty": 3}))
    ddb.put_item(TableName="api", Item={"pk": {"S": "TENANT#t-42"}, "sk": {"S": "IDEMP#k-2"}, "status": {"S": "IN_PROGRESS"},
                                         "hash": {"S": handler.hashlib.sha256(json.dumps({"sku": "B", "qty": 1}).encode()).hexdigest()},
                                         "lock_until": {"N": str(int(time.time()) + 30)}, "expires_at": {"N": "0"}})
    print("busy:    ", call("k-2", {"sku": "B", "qty": 1}))
    ddb.update_item(TableName="api", Key={"pk": {"S": "TENANT#t-42"}, "sk": {"S": "IDEMP#k-2"}},
                    UpdateExpression="SET lock_until = :o", ExpressionAttributeValues={":o": {"N": "1"}})
    print("stale:   ", call("k-2", {"sku": "B", "qty": 1})[0], "(crashed claim taken over)")
    print("no key:  ", handler.handler({"requestContext": {"authorizer": {"tenant_id": "t"}}, "headers": {}}, None)["statusCode"])
    real = handler.ddb.transact_write_items
    handler.ddb.transact_write_items = lambda **kw: (_ for _ in ()).throw(TimeoutError("crash before commit"))
    try:
        call("k-3", {"sku": "C", "qty": 1})
    except TimeoutError as e:
        print("crash:   ", e)
    handler.ddb.transact_write_items = real
    ddb.update_item(TableName="api", Key={"pk": {"S": "TENANT#t-42"}, "sk": {"S": "IDEMP#k-3"}},
                    UpdateExpression="SET lock_until = :o", ExpressionAttributeValues={":o": {"N": "1"}})
    print("recover: ", call("k-3", {"sku": "C", "qty": 1})[0], "(lock expired, same deterministic order_id)")
    orders = ddb.query(TableName="api", KeyConditionExpression="pk = :p AND begins_with(sk, :o)",
                       ExpressionAttributeValues={":p": {"S": "TENANT#t-42"}, ":o": {"S": "ORDER#"}})["Count"]
    msgs = sqs.get_queue_attributes(QueueUrl=q, AttributeNames=["ApproximateNumberOfMessages"])["Attributes"]
    print("orders written:", orders, "| queued jobs:", msgs["ApproximateNumberOfMessages"], "(k-3 queued twice, same order_id)")

    # Worker: a duplicate delivery is skipped, a failing job is reported alone.
    os.environ["TABLE_NAME"] = "api"
    import worker
    oid = first[1]["order_id"]
    batch = {"Records": [{"messageId": "m1", "body": json.dumps({"tenant": "t-42", "order_id": oid})},
                         {"messageId": "m2", "body": json.dumps({"tenant": "t-42", "order_id": oid})},
                         {"messageId": "m3", "body": json.dumps({"tenant": "t-42", "order_id": "x", "sku": "BROKEN"})}]}
    print("worker:  ", worker.handler(batch, None))

text

first:    (202, {'order_id': 'ef3cac65-23a0-5256-b6c4-a2f0640f7dc4', 'status': 'accepted'})
retry:    (202, {'order_id': 'ef3cac65-23a0-5256-b6c4-a2f0640f7dc4', 'status': 'accepted'}) same order: True
misuse:   (422, {'error': 'Idempotency-Key was already used with a different body'})
busy:     (409, {'error': 'a request with this key is still in progress; retry shortly'})
stale:    202 (crashed claim taken over)
no key:   400
crash:    crash before commit
recover:  202 (lock expired, same deterministic order_id)
orders written: 3 | queued jobs: 4 (k-3 queued twice, same order_id)
worker:   {'batchItemFailures': [{'itemIdentifier': 'm3'}]}

The retry returns the same order ID, the misuse is rejected, the stale claim is taken over, and after the simulated crash the recovery writes the same order: three orders for three keys. The crashed key was queued twice with the same order_id, which is why the worker has to dedupe. If you’d rather not write this yourself, Powertools for AWS Lambda has an idempotency utility built on the same idea (records with INPROGRESS and COMPLETE states in DynamoDB, with an expiry).

Step 2: A worker that tolerates duplicates and partial failures

Standard SQS queues deliver at least once, so the worker checks whether a job is done before doing it, and marks it done only after it succeeds. With ReportBatchItemFailures on the event source mapping, the function returns only the failed message IDs in batchItemFailures, so one bad message doesn’t make the whole batch retry.

python

"""SQS worker: skip jobs already done (delivery is at least once), report only the failed messages."""
import json
import os

import boto3

ddb = boto3.client("dynamodb")


def handler(event, context):
    table, failures = os.environ["TABLE_NAME"], []
    for rec in event["Records"]:
        job = json.loads(rec["body"])
        key = {"pk": {"S": f"TENANT#{job['tenant']}"}, "sk": {"S": f"JOB#{job['order_id']}"}}
        try:
            if "Item" in ddb.get_item(TableName=table, Key=key, ConsistentRead=True):
                continue                                    # duplicate delivery of a finished job
            fulfil(job)                                     # must itself be safe to repeat (e.g. keyed on order_id)
            ddb.put_item(TableName=table, Item=key)         # mark done only after it succeeded
        except Exception:
            failures.append({"itemIdentifier": rec["messageId"]})   # only this message is retried
    return {"batchItemFailures": failures}


def fulfil(job: dict) -> None:
    if job.get("sku") == "BROKEN":
        raise RuntimeError("downstream failed")

In the test output above, the duplicate delivery (m2) is skipped and only the failing job (m3) is reported. After five failed receives, the queue’s redrive policy moves it to the dead-letter queue. The fulfil step itself should still be safe to repeat, because a crash between doing the work and marking it done means one more attempt.

Step 3: The infrastructure

A SAM template with a regional REST API, stage throttling, the two functions, an on-demand table with TTL on the idempotency records, the queue with a dead-letter queue, and a WAF web ACL with a per-IP rate rule attached to the stage:

yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Orders API - REST API (regional) + Lambda + DynamoDB + SQS, with WAF and stage throttling.

Resources:
  Api:
    Type: AWS::Serverless::Api
    Properties:
      StageName: prod
      EndpointConfiguration: { Type: REGIONAL }
      TracingEnabled: true
      MethodSettings:
        - ResourcePath: "/*"
          HttpMethod: "*"
          ThrottlingRateLimit: 500        # steady requests per second for this stage
          ThrottlingBurstLimit: 1000
          MetricsEnabled: true

  OrdersFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Handler: handler.handler
      CodeUri: .
      MemorySize: 512
      Timeout: 10                         # well under the API Gateway integration timeout
      Environment:
        Variables: { TABLE_NAME: !Ref Table, QUEUE_URL: !Ref WorkQueue }
      Policies:
        - DynamoDBCrudPolicy: { TableName: !Ref Table }
        - SQSSendMessagePolicy: { QueueName: !GetAtt WorkQueue.QueueName }
      Events:
        Post:
          Type: Api
          Properties: { RestApiId: !Ref Api, Path: /orders, Method: post }

  Table:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - { AttributeName: pk, AttributeType: S }
        - { AttributeName: sk, AttributeType: S }
      KeySchema:
        - { AttributeName: pk, KeyType: HASH }
        - { AttributeName: sk, KeyType: RANGE }
      TimeToLiveSpecification: { AttributeName: expires_at, Enabled: true }
      PointInTimeRecoverySpecification: { PointInTimeRecoveryEnabled: true }

  WorkQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 180              # at least 6x the worker timeout
      RedrivePolicy: { deadLetterTargetArn: !GetAtt WorkDLQ.Arn, maxReceiveCount: 5 }
  WorkDLQ:
    Type: AWS::SQS::Queue
    Properties: { MessageRetentionPeriod: 1209600 }

  WorkerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Handler: worker.handler
      CodeUri: .
      Timeout: 30
      Environment:
        Variables: { TABLE_NAME: !Ref Table }
      Policies:
        - DynamoDBCrudPolicy: { TableName: !Ref Table }
      Events:
        Jobs:
          Type: SQS
          Properties:
            Queue: !GetAtt WorkQueue.Arn
            BatchSize: 10
            FunctionResponseTypes: [ReportBatchItemFailures]

  WebAcl:
    Type: AWS::WAFv2::WebACL
    Properties:
      Scope: REGIONAL
      DefaultAction: { Allow: {} }
      VisibilityConfig: { SampledRequestsEnabled: true, CloudWatchMetricsEnabled: true, MetricName: orders-api }
      Rules:
        - Name: per-ip-rate
          Priority: 0
          Action: { Block: {} }
          Statement:
            RateBasedStatement: { Limit: 2000, AggregateKeyType: IP }
          VisibilityConfig: { SampledRequestsEnabled: true, CloudWatchMetricsEnabled: true, MetricName: per-ip-rate }
  WebAclAssociation:
    Type: AWS::WAFv2::WebACLAssociation
    Properties:
      ResourceArn: !Sub arn:${AWS::Partition}:apigateway:${AWS::Region}::/restapis/${Api}/stages/prod
      WebACLArn: !GetAtt WebAcl.Arn
    DependsOn: ApiprodStage

Outputs:
  Endpoint:
    Value: !Sub https://${Api}.execute-api.${AWS::Region}.amazonaws.com/prod/orders

text

template.yaml: cfn-lint passed (SAM transform + WAF, SQS, DynamoDB)

A few settings deserve a note. The queue’s visibility timeout is six times the worker’s timeout, as the Lambda docs recommend for SQS event sources. The WAF rate limit counts requests per IP over the rule’s evaluation window, which defaults to 5 minutes. TTL removes old idempotency records automatically, but DynamoDB deletes expired items within a few days, not instantly. The stage throttle (500 requests per second, 1,000 burst) is an example; for per-client limits, add API keys and usage plans.

Worked example: sizing for a launch

Marketing expects 3,000 requests per second for an hour, and the handler averages 120 ms.

Concurrency: 3,000 × 0.12 = 360. That fits in the default 1,000, with room for the worker and other functions. Reserve concurrency for the worker so a spike in the API can’t starve it, or the other way round.
Scaling: 360 environments is well inside one 10-second scaling step of 1,000.
Gateway: 3,000 requests per second is under the 10,000 account default, but it’s shared with every other API in the account and Region. Check what else runs there.
Stage throttle: raise it from 500 to about 4,000 for the launch, so it still protects the backend from runaway clients.
DynamoDB on-demand handles the write rate, provided keys spread across tenants; one huge tenant on one partition key is the thing to watch.

Cost levers

Use an HTTP API if you don’t need REST API features; it’s offered at a lower price.
Keep handlers fast: Lambda bills duration, and slow handlers also use concurrency.
Cache cacheable GETs (REST API caching or a CloudFront distribution in front of a regional API).
On-demand DynamoDB for spiky traffic; consider provisioned capacity once traffic is steady and predictable.
Check current prices on the API Gateway, Lambda and DynamoDB pricing pages rather than trusting numbers in posts.

Troubleshooting

Symptom	Likely cause	Fix
Duplicate orders	No idempotency key, or a random ID per attempt	Claim the key conditionally; derive the ID from it
504 from API Gateway	Integration took longer than the timeout	Return 202 and move the work to SQS
429s during a spike	Stage, account or Lambda concurrency limit	Check which metric hit its limit; raise the right quota
Whole SQS batch retried for one bad message	No partial batch response	Enable `ReportBatchItemFailures`; return `batchItemFailures`
Messages processed twice	Visibility timeout shorter than processing time, or normal at-least-once delivery	Six times the function timeout; dedupe on a job key
Can’t attach WAF	Using an HTTP API	WAF attaches to REST API stages; switch, or filter elsewhere

FAQ

Do I need ElastiCache or OpenSearch from day one?

No. Add a cache when measurements show hot reads DynamoDB can’t serve cheaply, and a search index when you need queries DynamoDB can’t express. Each extra store is another consistency problem.

When should I move off Lambda?

When handlers are long-running or CPU-heavy, need long-lived connections, or when steady high throughput makes containers on ECS or EKS cheaper. Measure before switching.

Should the client or the server generate the idempotency key?

The client, once per logical operation, reused on every retry of that operation. A server-generated key can’t tell a retry from a new request.

Is 202 the right status for an accepted order?

Yes, when the work finishes asynchronously. Return the order ID so the client can poll or receive a webhook.

Official documentation

Last updated on October 10, 2026

Watch: 100 Days of AI on YouTube

Short videos from CheatCoders, one AI engineering topic per day.

Open the playlist on YouTube · CheatCoders on YouTube

Newly added

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.

Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Serverless API on AWS: API Gateway, Lambda, DynamoDB and SQS, From First Endpoint to Production

REST API or HTTP API?

The limits you’ll actually hit

Step 1: An idempotent write path

Step 2: A worker that tolerates duplicates and partial failures

Step 3: The infrastructure

Worked example: sizing for a launch

Cost levers

Troubleshooting

FAQ

Do I need ElastiCache or OpenSearch from day one?

When should I move off Lambda?

Should the client or the server generate the idempotency key?

Is 202 the right status for an accepted order?

Official documentation

Watch: 100 Days of AI on YouTube

Most viewed

Newly added

Like this:

Deep-dive PDF

Related

Discover more from CheatCoders

Comments

Leave a comment Cancel reply

REST API or HTTP API?

The limits you’ll actually hit

Step 1: An idempotent write path

Step 2: A worker that tolerates duplicates and partial failures

Step 3: The infrastructure

Worked example: sizing for a launch

Cost levers

Troubleshooting

FAQ

Do I need ElastiCache or OpenSearch from day one?

When should I move off Lambda?

Should the client or the server generate the idempotency key?

Is 202 the right status for an accepted order?

Official documentation

Watch: 100 Days of AI on YouTube

Share this:

Most viewed

Newly added

Like this:

Deep-dive PDF

Related

Discover more from CheatCoders

Comments

Leave a comment Cancel reply