ECR + Lambda Container Images: Heavyweight Coding Tools Without Zip Limits

0 views

Your run_tests / typecheck / parse_ast tool Lambda just hit the 250 MB unzipped wall — again. You deleted docs from node_modules, split layers, and still cannot ship clangd, a full TypeScript language service, or a CUDA-free torch wheel set. Lambda container images from ECR are the unfair advantage for heavyweight coding tools: package up to 10 GB, push to ECR, point the function at the image URI, keep the same event/IAM model. Pair with CodeBuild sandboxes for multi-minute builds and AWS Batch for overnight jobs — this post is the interactive heavy tool path.

⚡ TL;DR: Use Lambda container images when tool deps exceed Zip/layer limits. Multi-stage Dockerfiles, digests not :latest in prod, ECR scan on push, provisioned concurrency for hot tools. Prefer CodeBuild/Batch when wall clock > 15 minutes or you need a real VM. Related: CodeBuild sandboxes, Batch overnight jobs, SSM hierarchical config.

When Zip dies and containers win

Workload Zip/layers Container image CodeBuild / Batch
Small JSON transform tool Overkill No
tree-sitter + multi-lang grammars Painful Optional
Full LSP / clang toolchain ✅ (watch size/cold start) Better if >15m
Repo-wide migration overnight ❌ (15m cap) ✅ Batch
Untrusted npm test Risky Risky ✅ sandbox

Rule of thumb: container Lambda for heavy deps under 15 minutes and trusted code; CodeBuild/Batch for duration, isolation, or rootful package installs.

Minimal Dockerfile for a coding tool Lambda

Lambda base images include the Runtime Interface Client. Keep the image lean — every 100 MB hurts cold start.

dockerfile
# ✅ Multi-stage: build grammars, ship slim runtime
FROM public.ecr.aws/lambda/nodejs:20 AS build
WORKDIR /opt/build
COPY package*.json ./
RUN npm ci --omit=dev
COPY src ./src
RUN npm run build

FROM public.ecr.aws/lambda/nodejs:20
# Heavy native deps — pin versions; do not apt-get random PPAs in prod
COPY --from=build /opt/build/dist ${LAMBDA_TASK_ROOT}/
COPY --from=build /opt/build/node_modules ${LAMBDA_TASK_ROOT}/node_modules
# Example: prebuilt tree-sitter wasm / .node bindings only
COPY vendor/tree-sitter ${LAMBDA_TASK_ROOT}/vendor/tree-sitter
CMD ["handler.main"]
dockerfile
# ❌ Fat image anti-pattern
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y build-essential clangd nodejs npm python3 ...
# Missing Lambda RIC, multi-GB, unbounded CVE surface, slow pulls

For Python tools with large wheels:

dockerfile
FROM public.ecr.aws/lambda/python:3.12
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt -t ${LAMBDA_TASK_ROOT}
COPY app/ ${LAMBDA_TASK_ROOT}/
CMD ["app.handler.handler"]

Strip torch/CUDA if you only need tokenization; prefer CPU wheels. Put model weights in S3/EFS if they dominate size — do not bake 3 GB checkpoints into every revision.

Build, push, and pin by digest

bash
ACCOUNT=123456789012
REGION=us-east-1
REPO=agent-tools/typecheck
IMAGE_TAG=2026-09-22.1

aws ecr create-repository --repository-name "$REPO" --image-scanning-configuration scanOnPush=true

docker build -t "$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:$IMAGE_TAG" .
aws ecr get-login-password --region "$REGION" \
  | docker login --username AWS --password-stdin "$ACCOUNT.dkr.ecr.$REGION.amazonaws.com"
docker push "$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:$IMAGE_TAG"

# Resolve digest for immutable function config
DIGEST=$(aws ecr describe-images --repository-name "$REPO" \
  --image-ids imageTag=$IMAGE_TAG \
  --query 'imageDetails[0].imageDigest' --output text)
echo "Pin: $ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO@$DIGEST"
typescript
// ✅ CDK/CFN: ImageUri with digest; never :latest in prod
const fn = new lambda.DockerImageFunction(this, "TypecheckTool", {
  code: lambda.DockerImageCode.fromEcr(repo, { tagOrDigest: digest }),
  memorySize: 3008,
  timeout: Duration.seconds(120),
  architecture: lambda.Architecture.ARM_64, // often cheaper/faster for Node
});

Store the digest in SSM Parameter Store per env (/agents/prod/tools/typecheck/image_digest) so rollbacks are a parameter change + function update, not archaeology.

Cold starts, concurrency, and security

Container images increase pull + init time. Mitigations:

  1. Provisioned concurrency on the hottest tools (typecheck, lint) — pay for warmth.
  2. ARM64 when deps support it — often better price/perf.
  3. Slim base + no shell tooling in the final stage.
  4. ECR enhanced scanning + block deploy on CRITICAL.
  5. Least-privilege IAM on the tool role; network egress via VPC only if required (VPC adds cold-start cost).
bash
# ❌ Mutating :latest in place and hoping all aliases pick it up
docker push .../typecheck:latest
# Incident: half of concurrency fleet on old layers, half on new

Authorize who may invoke the tool with Verified Permissions and rate-limit public gateways with API Gateway + WAF. Container size is not a sandbox — untrusted user code still belongs in CodeBuild/Firecracker-style isolation, not your shared tool image.

Decision checklist vs Batch / CodeBuild

  • [ ] Dep size > Zip limits but runtime usually < 15 minutes → container Lambda.
  • [ ] Need > 15 minutes or multi-GB scratch disk → Batch or CodeBuild.
  • [ ] Untrusted code execution → CodeBuild sandbox / Fargate, not shared Lambda image.
  • [ ] Image pinned by digest; scanOnPush on; CI fails on CRITICAL CVEs.
  • [ ] Provisioned concurrency for p95-sensitive tools.
  • [ ] Digests recorded in SSM; rollback documented.
  • [ ] Memory tuned (CPU scales with memory on Lambda) — profile AST parse CPU.
  • [ ] Observability: ADOT span tool.image_digest.

Heavyweight coding tools should not force you into Zip diets or premature ECS sprawl. ECR + Lambda container images buy you the toolchain; discipline on digests, scanning, and duration boundaries keeps them production-grade.

CI pattern that will not surprise prod

Wire your agent-tools monorepo so every Dockerfile change produces an immutable tag + digest artifact:

  1. Buildx with --platform linux/arm64 (or dual arch if you must).
  2. Push tag gitsha and date.buildN.
  3. Run ECR wait for scan findings; fail pipeline on CRITICAL/HIGH as policy.
  4. Publish digest to SSM /agents/${env}/tools/${name}/image_digest.
  5. Deploy function update referencing digest only.
  6. Smoke-invoke with a golden fixture repo tarball in S3.
  7. Only then flip AppConfig percentage traffic if you use gradual tool rollout.

Local sam local / Docker RIC testing catches CMD/handler mismatches before you burn ECR storage. Keep a Makefile target tool-smoke that the Step Functions canary also calls.

Handler sketch for a typecheck tool

typescript
import { execFile } from "node:child_process";
import { promisify } from "node:util";
const execFileAsync = promisify(execFile);

export async function main(event: {
  tenant_id: string;
  session_id: string;
  workdir_s3: string;
}) {
  // Download workdir from S3 (omit); never trust path traversal
  const workdir = "/tmp/work";
  try {
    const { stdout, stderr } = await execFileAsync(
      "npx",
      ["--no-install", "tsc", "-p", `${workdir}/tsconfig.json`, "--pretty", "false"],
      { timeout: 90_000, maxBuffer: 2 * 1024 * 1024 }
    );
    return { ok: true, stdout, stderr };
  } catch (err: any) {
    // tsc exits non-zero on type errors — that is a tool result, not infra failure
    if (err.code === 1) {
      return { ok: false, stdout: err.stdout, stderr: err.stderr, error_class: "type_errors" };
    }
    throw err;
  }
}

Treat compiler exit code 1 as a structured tool result for the planner, not a Lambda failure that burns retries and Destinations noise.

Deep-dive PDF

Get the expanded guide for this post — extra diagrams-style checklists, failure modes, and a production walkthrough. Free when you subscribe to CheatCoders.

Already subscribed? or open the subscribe page.


Discover more from CheatCoders

Subscribe to get the latest posts sent to your email.

Comments

No comments yet. Why don’t you start the discussion?

Leave a comment

No account needed. Name and email are optional.