Designing an API gateway layer for tens of millions of users

A single API gateway instance cannot serve a national telecom's traffic. Neither can a gateway that calls a database on every request, or one that is the only place you can stop an attack. This post walks through the shape of a gateway layer that holds up when tens of millions of people use the same apps every day — the design I keep coming back to after leading a telecom platform from monoliths to microservices.

Start at the edge, not at the gateway

The first hop for every request — mobile app or browser — should be an edge network, not your cluster. I use Cloudflare for this. Before a request ever reaches your infrastructure, the edge can:

• Drop known-bad traffic with a WAF and managed rules.
• Apply rate limits per IP, path, or header, so a misbehaving client never costs you a pod.
• Absorb DDoS volumes your cluster could never scale for.
• Run small bits of logic in Workers — redirects, header normalisation, geo rules.

The rule of thumb: anything that can be rejected without knowing who the user is should be rejected at the edge. Your gateway's CPU is for the requests that matter.

The gateway is a stateless fleet, not a box

Inside Kubernetes, traffic enters through an ingress or load balancer and spreads across many gateway pods. Every pod is identical and keeps no session state, which means:

• A Horizontal Pod Autoscaler can add pods when CPU or request rate climbs, and remove them at night.
• Any pod can die at any moment, and the load balancer simply stops sending it traffic.
• Rolling deploys replace pods a few at a time with no downtime.

Each pod does the same three things, in this order:

1. Rate-limit by user or tenant — the edge limited by IP; the gateway limits by identity.
2. Verify the JWT — signature, expiry, audience, and tenant.
3. Proxy the request to the service that owns it, and stream the response back.

That is all. Business logic lives in services, not in the gateway.

Verify tokens without calling anyone

The most common gateway mistake is calling the identity service to validate every token. At this scale that turns your IAM service into the bottleneck for the entire platform.

Instead, the IAM service signs short-lived JWTs with a private key and publishes the matching public keys as a JWKS. Each gateway pod:

• Fetches the JWKS from the IAM service once at boot.
• Keeps it in memory for verification, so checking a token is pure CPU — no network call.
• Shares it through Redis, so a newly started pod has keys even if the IAM service is briefly unavailable.
• Refreshes when it sees a kid (key ID) it doesn't know, which is how key rotation works without a deploy.

// Verify a token with keys already in memory. No network call on the hot path.
func (v *Verifier) Verify(raw string) (*Claims, error) {
	claims := &Claims{}
	_, err := jwt.ParseWithClaims(raw, claims, func(t *jwt.Token) (any, error) {
		kid, _ := t.Header["kid"].(string)
		if key, ok := v.keys.Load(kid); ok {
			return key, nil
		}
		return v.refreshAndGet(kid) // unknown kid: rotation happened, reload JWKS once
	}, jwt.WithValidMethods([]string{"RS256"}), jwt.WithAudience(v.audience))
	if err != nil {
		return nil, err
	}
	return claims, nil
}

A gateway that verifies tokens locally scales with the number of pods. A gateway that asks IAM scales with IAM.

Every service owns its data

Behind the gateway, each service — IAM, user access management, profile, bonus, campaigns — owns its own database. The gateway never queries them directly, and services don't reach into each other's tables. Some services need more than one store: a profile service, for example, keeps the source of truth in its database and hot profile reads in Redis.

This is what makes the platform changeable. A team can migrate its schema, change its cache, or rewrite its service in another language, and nobody else has to coordinate a release.

Push slow work out of the request path

Some work should never block a user's request: sending SMS offers, crediting a bonus, writing an audit trail. Put it on a queue (RabbitMQ, SQS) and let a worker do it. A campaign service that sends SMS through a third-party provider is the classic example — the provider's latency and outages become the worker's problem, retried with backoff, instead of the user's.

What I'd check first in any gateway design

• Can you reject abusive traffic before it reaches your cluster?
• Can any gateway pod die right now without users noticing?
• Does verifying a token require a network call? If yes, fix that first.
• Does any service read another service's database?
• Which requests wait on a third party, and could they go on a queue instead?

Get those five right and the rest of the system has room to grow.