To appreciate the latency budget of a gateway, one must follow a request packet as it traverses the internal "Stages." A modern gateway like Envoy or Kong doesn't just "forward" bits; it deconstructs and reconstructs the world in nanoseconds.

Traditionally, extending a gateway meant writing Lua scripts (NGINX/Kong) or recompiling the binary (Envoy). In 2026, **WebAssembly (WASM)** has become the universal standard for gateway extensibility.

A common architectural mistake is confusing the **API Gateway** (Ingress) with a **Service Mesh** (North-South vs East-West). In 2026, the lines have blurred, but the forensic distinction remains critical for security audits.

"The Gateway protects the cluster from the Internet; the Mesh protects the services from each other."

When using API Gateways with Serverless backends (AWS Lambda, Google Cloud Functions), the gateway's role shifts from a static forwarder to a complex **Connection Manager**.

Connection pooling between the API Gateway and its upstream services is one of the most impactful performance tuning levers available to platform engineers. Every new TCP connection to an upstream service requires a three-way handshake (1.5 RTT) and, if TLS is used, an additional 1-2 RTT for the TLS handshake. For a gateway handling 100,000 requests per second spread across 50 upstream services, the connection establishment overhead alone can consume 15-20% of the gateway's CPU if connections are not pooled and reused effectively.

Envoy's connection pool architecture is the de facto standard for modern API gateways. Each worker thread maintains its own connection pool per upstream cluster, per priority level. The pool parameters govern three critical behaviors: **max_connections** (the hard limit on the number of TCP connections to a single upstream host), **max_pending_requests** (the number of requests that can queue while waiting for a connection), and **max_requests_per_connection** (the number of requests a single connection can serve before being closed). The `max_connections` parameter is the most sensitive: setting it too low causes request queuing and increased latency; setting it too high overwhelms the upstream with connection churn. The recommended starting value is `max_connections = (expected RPS × P99 latency) / concurrency_per_connection`. For a service handling 1,000 RPS with 100ms P99 latency and 10 requests per connection, the optimal max_connections is 10.

Keep-alive timeout tuning is the second critical parameter. The HTTP/1.1 keep-alive timeout determines how long an idle connection stays open before being closed. A short timeout (5-10 seconds) ensures that connections are released quickly during traffic troughs but causes frequent reconnections during traffic bursts. A long timeout (60-300 seconds) maximizes connection reuse but holds idle connections open, consuming memory and TCP resources on both the gateway and the upstream. The optimal keep-alive timeout is approximately 3x the average inter-request arrival time. For a service receiving 100 requests per minute, the average inter-arrival time is 600ms, making the optimal keep-alive timeout approximately 1.8 seconds. In practice, most deployments use a default of 30-60 seconds, which works well for services with request rates above 2 requests per second.

Envoy's **connection draining** behavior during upstream pod termination is a critical operational concern. When a Kubernetes pod enters the `Terminating` state, Envoy must stop sending new requests to that pod and allow existing requests to complete gracefully. Envoy handles this through **active health checking** — if health checking is enabled (recommended interval: 1 second, with 2-3 failure threshold), Envoy detects the pod's transition to unhealthy and removes it from the load balancing set. However, there is a race condition: if the pod's `preStop` hook executes `sleep 15` before `SIGTERM`, but Envoy's health check interval is 3 seconds, there are 12 seconds where the pod is still receiving traffic. The solution is **outlier detection** combined with **drain detection**: Envoy monitors the upstream's connection close behavior and, if it detects a pattern of connection resets from a specific host, proactively ejects it from the load balancing set. This automatic detection reduces the connection error rate during rolling updates from 2-5% to less than 0.1%.

HTTP/2 connection pooling differs significantly from HTTP/1.1. In HTTP/2, a single connection supports up to 100 concurrent streams (configurable via `MaxConcurrentStreams`). The `max_connections` per upstream host should therefore be much lower — typically 1-4 connections instead of 10-50 for HTTP/1.1. However, the connection sensitivity is higher: if the single HTTP/2 connection is lost (due to a network partition or upstream restart), all 100 in-flight streams fail simultaneously, creating a traffic spike on the remaining connections. The recommended HTTP/2 configuration for production is 2-3 connections per upstream host, with `MaxConcurrentStreams` set to match the upstream service's concurrency capacity. This provides both high multiplexing efficiency (up to 300 concurrent streams) and connection-level redundancy (if one connection fails, 200 streams continue uninterrupted).

Request aggregation at the API Gateway level is a powerful pattern for reducing client-side complexity and improving perceived performance. Instead of requiring a mobile app or web frontend to make 5-10 separate API calls to assemble a single page, the gateway accepts one request, fans it out to multiple upstream services in parallel, merges the responses, and returns a single unified payload. This reduces the number of round trips from N to 1 and moves the aggregation logic from the client (uncontrolled, many versions) to the gateway (controlled, one version).

The aggregation request flow follows a scatter-gather pattern. The gateway receives an incoming request, parses it to extract the required data fields, creates N parallel sub-requests to the upstream services, waits for all responses (or a configurable timeout), and merges them into a single response. The latency of the aggregated endpoint is determined by the slowest upstream service, not the sum of all services. If the upstream services have independent latency distributions, the aggregated P99 latency is approximately P99_max × sqrt(log N) — the square root of the log of the number of services. For 9 upstream services with independent P99 latencies of 100ms, the aggregated P99 is approximately 100ms × sqrt(log 9) = 100ms × 1.48 = 148ms, compared to 900ms if the requests were sequential.

The response merging phase is where the performance bottleneck shifts from network latency to CPU-bound serialization. The gateway must parse each upstream response (typically JSON or Protobuf), extract the relevant fields, and merge them into a new response structure. For a merge of 10 upstream responses, each 50KB in size, the gateway must parse 500KB of JSON — a non-trivial operation that can consume 2-5ms of CPU time per request. At 10,000 requests per second, this translates to 20-50 seconds of CPU time per second, requiring 20-50 CPU cores dedicated solely to response merging. The optimization is **schema-aware partial deserialization**: instead of fully parsing all JSON fields, the gateway only deserializes the fields that are needed for the merged response. A schema-aware gateway can reduce parsing time by 60-80% by skipping unnecessary fields, dropping the CPU requirement for response merging from 50 cores to 10-15 cores for the same throughput.

Partial failure handling is the most complex aspect of request aggregation. If one of the 10 upstream services fails (returns 5xx or times out), the gateway must decide: fail the entire request, or return a partial response with an error indicator for the failed service. The **graceful degradation** approach returns a partial response with a 200 status code and an embedded error object describing the failed service. This is the correct approach for read-heavy UIs where a missing section is acceptable (e.g., "recommendations unavailable" in an e-commerce page). The **fail-fast** approach returns a 5xx error for the entire request, which is appropriate when the failed data is critical (e.g., the user's account balance in a banking app). The gateway implements this decision through per-service error policies configured in the route definition — a single aggregated endpoint can have mixed policies where critical services use fail-fast and non-critical services use graceful degradation.

The performance of request aggregation can be optimized through **response caching at the gateway level**. Cache-Control headers from upstream responses inform the gateway's cache behavior. If an upstream response for "product recommendations" has a `max-age=60` header, the gateway can cache that response for 60 seconds and serve it to subsequent aggregated requests without calling the upstream service. In a benchmark with 5 upstream services, caching reduced the aggregated endpoint's P99 latency from 180ms to 45ms for cached responses — a 75% reduction. The cache hit ratio depends on the content popularity distribution and the cache size. A 1GB in-memory cache at the gateway can store approximately 20,000 aggregated response fragments, providing an 85-95% hit ratio for most e-commerce workloads. Cache invalidation is handled through Cache-Control stale-while-revalidate headers, which allow the gateway to serve stale data while asynchronously refreshing the cache in the background.