API gateway and service mesh architecture
Procurement conversations often treat "gateway" and "mesh" as competing products. The architecture question is about the direction of traffic. A gateway manages traffic entering a system from outside. A mesh manages traffic between the services already inside. Most systems that outgrow a plain load balancer eventually need something like both, for two different reasons.
1. North-south versus east-west traffic
North-south traffic crosses a system's outer boundary: a client on the internet calling an API. East-west traffic runs between services inside that boundary: the checkout service calling the inventory service, which calls the pricing service. They're different problems. North-south traffic is relatively low in volume, arrives from untrusted clients, and needs one well-defined entry point where authentication and rate limiting are enforced once. East-west traffic is relatively high in volume, since one external request commonly fans out into several internal calls, and it comes from services nominally inside the trust boundary. What matters there is keeping one service's slowness or failure from cascading into every service that calls it.
2. What an API gateway does
A gateway sits at the system's edge. External traffic authenticates there, gets rate limited there, and is routed to whichever internal service owns the endpoint. Centralizing this means internal services don't each reimplement authentication or throttling, and a policy change (a new rate limit tier, a revoked API key) takes effect in one place instead of being pushed to every service with a public endpoint. Gateways also commonly translate protocols: accepting a REST or GraphQL request at the edge and dispatching it as an internal gRPC call, or aggregating several internal calls into the single response shape an external client wants, so clients never need to know the internal service topology.
3. What a service mesh does
A mesh handles the east-west problem: reliability and observability for calls between services that already trust each other. A gateway at the edge never sees this volume or these failure modes, because it's in the path only for the first hop of a request, not the four internal hops that may follow. The standard mesh feature set: mutual TLS between services, so every internal call is authenticated and encrypted without each service implementing it; automatic retries with backoff for a transiently failing service; circuit breaking that stops sending traffic to a clearly unhealthy downstream service instead of piling on more failing requests; and consistent tracing and metrics for every internal call, whichever team wrote the service.
The mechanism is the sidecar: a small proxy deployed alongside each service instance (in Kubernetes, a second container in the same pod) that transparently intercepts all inbound and outbound traffic and applies mesh policy at the network layer instead of in the service's code. That's how mesh features apply uniformly across services written in different languages by different teams. The policy lives in the proxy, so no team has to import and maintain a library.
4. Where they overlap, and where the line is
Both do routing, both can enforce policy, and both produce telemetry, which is why they get pitched as substitutes. The line that matters is scope and trust boundary. A gateway's job ends once a request is authenticated and routed at the edge. A mesh's job starts there and covers every internal hop that follows, none of which pass through the gateway. A gateway with no mesh behind it can't see, retry, or circuit-break a failure three hops deep in a call chain it left after the first hop. A mesh with no gateway in front has no central place to authenticate an external client before its request reaches an internal service.
5. The sidecar tradeoff
The sidecar-per-instance model has costs. Every service instance runs an extra proxy process with its own CPU and memory, and every internal call makes two extra network hops, through the caller's sidecar and the receiver's, that a direct call wouldn't. With many services at high request volumes, that overhead is a measurable tax, paid on every internal call whether or not that call needed retries, circuit breaking, or mTLS.
A mesh is also complex to operate: mTLS certificate rotation across every instance, a control plane that must stay healthy for policies to apply consistently, and a new recurring incident question, "is this failure the service's logic or a mesh policy intercepting the call?" Where a mesh solves a problem you have, pay these costs. Don't adopt one because it appears on an architecture diagram template before you have the service count and failure-mode pain a mesh exists to fix.
6. When you need neither
A system with a handful of services, modest internal call volume, and a request graph simple enough to reason about without dedicated tracing and circuit-breaking doesn't need a mesh. A well-designed HTTP client with retry and timeout logic, applied consistently by convention or a shared library, covers the same ground at a fraction of the operational cost. An occasional transient failure between two or three services doesn't call for infrastructure built to coordinate reliability policy across dozens of independently deployed services. A system with no public API (an internal batch pipeline with no external clients) has no use for a gateway, because there's no untrusted edge traffic to sit in front of.
Add a gateway when there's an edge to protect and policy to centralize. Add a mesh when the number of services and the volume of internal calls make per-service retry logic and ad hoc tracing unmanageable. Don't add either on a schedule set by how mature the architecture diagram is supposed to look.
7. Worked example: two different problems, two different fixes
A team's checkout flow fails intermittently and is hard to diagnose. The external checkout endpoint returns errors at an elevated but not constant rate, and the on-call engineer can't tell from the gateway's logs whether the failure starts in the checkout service or further downstream. Adding a mesh gives every internal hop (checkout to inventory, inventory to the warehouse system) its own trace span and circuit breaker. The next incident shows the warehouse system timing out intermittently and inventory retrying without backoff, amplifying load on an already struggling service. That's an east-west reliability problem, which is what a mesh is for.
Separately, the team wants to let a few partner companies call a subset of its API directly, each with its own rate limit and API key, without every internal service implementing key validation and throttling. That's a north-south problem. A gateway in front of the system authenticates each partner's key once at the edge and applies that partner's rate limit before the request reaches any internal service. A mesh would contribute nothing here, because no external partner ever calls an internal service directly.
See microservices vs. monolith for whether a system's service count justifies either piece of infrastructure, and container orchestration: Kubernetes architecture fundamentals for the sidecar mechanism a mesh is typically built on.