Kong API Gateway and Service Mesh: A Complete Engineering Guide

Introduction

Every moderately complex distributed system eventually develops the same set of problems: you need consistent authentication across dozens of services, you need rate limiting that doesn't live in application code, you need traffic observability without adding instrumentation to every repository, and you need the ability to route, transform, and protect traffic without redeploying services. The API gateway pattern is a decades-old architectural answer to these concerns. Kong is one of the most widely deployed open-source implementations of that pattern.

Kong started as a Lua-based proxy built on top of Nginx, evolved into a platform with a rich plugin ecosystem, and has since expanded into the service mesh space with Kong Mesh (based on Kuma and Envoy). Understanding when and how to use Kong - and where its boundaries lie - is a practical skill for any engineer operating infrastructure at scale. This post provides a thorough, opinionated, production-oriented walkthrough: the concepts, the code, the configuration, the gotchas, and the decision framework.

The Problem Space: Why APIs and Meshes Exist

The Cross-Cutting Concern Problem

In a monolith, cross-cutting concerns - authentication, logging, rate limiting, tracing - are handled once, typically via middleware. When you decompose that monolith into microservices, each service needs to re-implement or import those concerns independently. At five services, this is manageable. At fifty, it becomes a maintenance crisis. You end up with inconsistent token validation logic across teams, ad-hoc retry policies baked into service code, and no single place to enforce a security policy change across the fleet.

The API gateway centralizes these concerns at the edge: the single point through which external traffic enters the system. The service mesh, by contrast, addresses the same problem for east-west traffic - the service-to-service communication inside the cluster. Together they form a complete policy enforcement layer without requiring application code changes.

The Observability Gap

Distributed systems are hard to observe. A single user request might fan out across eight services. Without a consistent tracing context injected at the gateway and propagated through every hop, debugging latency issues or error cascades requires correlating logs from multiple systems manually. The gateway is the natural injection point for trace IDs and correlation headers. The service mesh ensures those headers propagate through mTLS-authenticated, proxied connections even when the application developer forgets to forward them.

This is a concrete motivation, not an abstract architecture concern. Teams that instrument their gateway early find incident response times drop significantly because they can trace a failing request end-to-end from the gateway logs outward, rather than asking service owners to search their logs independently.

Security at the Perimeter and at the Network Layer

External-facing APIs need authentication, authorization, input validation, and DDoS mitigation. Those all belong at the gateway. But lateral movement inside a cluster - where a compromised service can freely call any other service - is a separate threat model. The service mesh addresses this with mutual TLS (mTLS) between every service pair, enforcing that only explicitly permitted services can communicate, and that all traffic is encrypted in transit regardless of the underlying network.

Neither the gateway alone nor the mesh alone is sufficient. You need both layers, with clear responsibility boundaries between them.

What Is Kong? Architecture and Core Concepts

Kong's Core Architecture

Kong is a cloud-native API gateway built on top of OpenResty (Nginx + LuaJIT). At its core, Kong is a reverse proxy that adds a plugin execution engine to the Nginx request-processing lifecycle. When a request arrives, Kong matches it against a configured Route, which belongs to a Service (the upstream), and executes a chain of Plugins during that request's lifecycle phases.

The key architectural entities are:

Service: An abstraction representing an upstream API (e.g., http://users-service:8080).
Route: A matching rule (host, path, method, headers) that maps to a Service.
Plugin: A unit of logic executed at one or more phases of the request/response lifecycle (access, header_filter, body_filter, log).
Upstream: A load-balancing pool of targets (instances of a service), supporting health checks and multiple algorithms.
Consumer: A representation of an API client, used as the subject for authentication and rate limiting policies.

Kong stores configuration in PostgreSQL (or in DB-less mode, declarative YAML/JSON). Kong Gateway 3.x introduced a hybrid mode where a control plane (CP) manages configuration and data plane (DP) nodes handle traffic, decoupling deployment concerns and enabling large-scale horizontal scaling of the data path without database access on the hot path.

DB-less and Declarative Configuration

One of the most operationally significant features introduced in Kong 1.1 is the DB-less mode, where the entire gateway configuration is expressed as a declarative YAML file (kong.yaml) and loaded at startup or via the Admin API's /config endpoint. This is essential for GitOps workflows: your gateway configuration lives in version control, goes through code review, and is deployed via CI/CD - no manual curl commands to the Admin API in production.

# kong.yaml - declarative configuration example
_format_version: "3.0"

services:
  - name: users-service
    url: http://users-svc:8080
    routes:
      - name: users-route
        paths:
          - /api/v1/users
        strip_path: false
    plugins:
      - name: jwt
        config:
          secret_is_base64: false
      - name: rate-limiting
        config:
          minute: 100
          policy: redis
          redis_host: redis
          redis_port: 6379

  - name: products-service
    url: http://products-svc:8080
    routes:
      - name: products-route
        paths:
          - /api/v1/products
        methods:
          - GET
          - POST
    plugins:
      - name: key-auth
      - name: prometheus

This declarative format makes configuration auditable and reproducible. The tradeoff is that DB-less mode historically had limitations around certain plugin configurations (e.g., OAuth2 flows that require runtime state writes), though Kong 3.x significantly narrowed this gap.

Kong's Plugin Lifecycle

Plugins execute in phases that correspond to Nginx's processing stages. Understanding this model is critical for writing custom plugins and for reasoning about plugin ordering:

init_worker: Runs once when a worker process starts. Used for background timers and cached data initialization.
access: Runs before proxying the request. Most policy enforcement happens here (auth, rate limiting, request transformation).
header_filter: Runs when the upstream response headers arrive. Used to add/remove response headers.
body_filter: Runs for each chunk of the response body. Used for response transformation (costly - avoid unless necessary).
log: Runs after the response is sent. Used for asynchronous logging to external systems.

Plugin priority determines execution order within a phase. Kong's bundled plugins have fixed priorities. Custom plugins can specify their own priority value. When two plugins execute in the same phase, the one with the higher priority number runs first.

Service Mesh Fundamentals and Where Kong Fits

The Sidecar Proxy Model

A service mesh works by injecting a sidecar proxy (typically Envoy) alongside every service instance. All inbound and outbound traffic for a service passes through its local sidecar rather than going directly to the network. The collection of sidecars forms the data plane; a centralized control plane (Istio's Istiod, Kuma's control plane, Linkerd's control plane) distributes configuration to those sidecars.

This architecture enables the mesh to enforce mTLS transparently, collect telemetry, implement circuit breaking, and apply traffic policies without any application code changes. The application thinks it is making a plain TCP connection to a local address; the sidecar handles encryption, retries, and observability silently.

Kong Mesh vs. Kong Gateway: Different Layers, Different Concerns

Kong offers two distinct products that are often conflated:

Kong Gateway (open source and enterprise) is an edge gateway - it handles north-south traffic (external clients to internal services). It terminates TLS, authenticates external consumers, rate-limits traffic, and routes requests to upstream services.

Kong Mesh is a service mesh built on top of Kuma (a CNCF project maintained by Kong) and uses Envoy as the data plane proxy. It handles east-west traffic (service-to-service) with mTLS, traffic policies, and observability.

These are complementary, not competing. In a fully realized architecture, Kong Gateway sits at the ingress boundary, and Kong Mesh governs communication between services inside the cluster. They share a policy model through Kuma's MeshTrafficPermission and related resources, which creates a consistent operational experience across both traffic directions.

When You Need a Mesh vs. Just a Gateway

The gateway alone is sufficient when:

All inter-service communication can be treated as trusted (e.g., isolated VPC, small team, low compliance requirements).
You have fewer than ~10 services and operational overhead of a mesh isn't justified.
East-west traffic policy is handled at the network layer (security groups, Kubernetes NetworkPolicy).

The mesh becomes necessary when:

Compliance requirements (PCI-DSS, HIPAA, SOC 2) mandate encryption of all in-cluster traffic.
You need fine-grained, auditable service-to-service authorization (zero-trust networking).
You want circuit breaking and retries without modifying service code.
Traffic observability between services is a production requirement, not a nice-to-have.

The decision is primarily driven by your threat model and compliance posture, not by technical complexity alone.

Kong with Docker: A Practical PoC

Architecture of the Docker PoC

The goal of this PoC is to run Kong in DB-less mode with two upstream services, demonstrate rate limiting, JWT authentication, and request logging, and explore the Admin API - all locally with Docker Compose. This setup mirrors a real development environment and is suitable for testing plugin configurations before promoting to Kubernetes.

# docker-compose.yml
version: "3.8"

services:
  kong:
    image: kong:3.6
    environment:
      KONG_DATABASE: "off"
      KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yaml
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: "0.0.0.0:8001"
      KONG_PLUGINS: "bundled"
      KONG_LOG_LEVEL: info
    volumes:
      - ./kong-config:/kong/declarative
    ports:
      - "8000:8000"   # proxy HTTP
      - "8443:8443"   # proxy HTTPS
      - "8001:8001"   # Admin API HTTP
    depends_on:
      - redis
    networks:
      - kong-net
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10

  redis:
    image: redis:7-alpine
    networks:
      - kong-net

  # Mock upstream: returns request details - useful for inspecting headers Kong adds
  httpbin:
    image: kennethreitz/httpbin
    networks:
      - kong-net

  # Second upstream simulating a product catalog service
  mock-api:
    image: mockserver/mockserver:latest
    environment:
      MOCKSERVER_INITIALIZATION_JSON_PATH: /config/mockserver.json
    volumes:
      - ./mock-api-config:/config
    networks:
      - kong-net

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - kong-net
    ports:
      - "9090:9090"

networks:
  kong-net:
    driver: bridge

Declarative Configuration for the PoC

# kong-config/kong.yaml
_format_version: "3.0"
_transform: true

consumers:
  - username: api-client-1
    jwt_secrets:
      - algorithm: HS256
        secret: "your-256-bit-secret-here"
    keyauth_credentials:
      - key: "client-one-api-key"

services:
  - name: inspection-service
    url: http://httpbin/
    connect_timeout: 5000
    read_timeout: 15000
    write_timeout: 15000
    routes:
      - name: inspect-route
        paths:
          - /inspect
        strip_path: true
    plugins:
      - name: jwt
        config:
          secret_is_base64: false
          claims_to_verify:
            - exp
      - name: rate-limiting
        config:
          minute: 30
          hour: 500
          policy: redis
          redis_host: redis
          redis_port: 6379
          fault_tolerant: true
      - name: correlation-id
        config:
          header_name: X-Correlation-ID
          generator: uuid#counter
          echo_downstream: true
      - name: prometheus

  - name: catalog-service
    url: http://mock-api:1080/
    routes:
      - name: catalog-route
        paths:
          - /catalog
        methods:
          - GET
        strip_path: false
    plugins:
      - name: key-auth
        config:
          key_names:
            - X-API-Key
          hide_credentials: true
      - name: request-transformer
        config:
          add:
            headers:
              - "X-Service-Version:v2"
              - "X-Forwarded-By:kong"
      - name: response-transformer
        config:
          remove:
            headers:
              - Server
              - X-Powered-By
      - name: prometheus

plugins:
  # Global plugin - applies to all routes
  - name: http-log
    config:
      http_endpoint: http://log-collector:3000/logs
      method: POST
      timeout: 3000
      keepalive: 60000
      flush_timeout: 2
      retry_count: 3

Validating the PoC

Once the stack is running, you can validate each concern:

# Start the stack
docker compose up -d

# Check Kong is healthy
curl -s http://localhost:8001/status | jq .

# Generate a JWT token for testing (using your secret)
# In a real setup, your auth service issues these
JWT_HEADER=$(echo -n '{"alg":"HS256","typ":"JWT"}' | base64 | tr -d '=\n')
JWT_PAYLOAD=$(echo -n "{\"iss\":\"api-client-1\",\"exp\":$(( $(date +%s) + 3600 ))}" | base64 | tr -d '=\n')
JWT_SIG=$(echo -n "${JWT_HEADER}.${JWT_PAYLOAD}" | openssl dgst -sha256 -hmac "your-256-bit-secret-here" -binary | base64 | tr '+/' '-_' | tr -d '=\n')
TOKEN="${JWT_HEADER}.${JWT_PAYLOAD}.${JWT_SIG}"

# Hit the protected route
curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/inspect/get

# Observe rate limiting headers in the response
curl -I -H "Authorization: Bearer $TOKEN" http://localhost:8000/inspect/get | grep -i "ratelimit"

# Check metrics (Prometheus endpoint)
curl http://localhost:8001/metrics | grep kong_http_requests_total

# Hit the key-auth route
curl -H "X-API-Key: client-one-api-key" http://localhost:8000/catalog/products

The X-Correlation-ID header will be present in every response, the Server header will be stripped, and after 30 requests within a minute from the same IP, you'll receive a 429 Too Many Requests with Retry-After headers set.

Kong with Kubernetes: Ingress, Gateway API, and the Mesh

Two Configuration Models: Ingress vs. Gateway API

Kong on Kubernetes can be configured via two Kubernetes-native mechanisms:

Ingress Controller model: Kong acts as the implementation for networking.k8s.io/v1 Ingress resources. Route configuration is expressed via Ingress annotations (konghq.com/plugins, konghq.com/strip-path, etc.). This model works but becomes annotation-heavy for complex configurations and couples Kong-specific logic to Kubernetes Ingress resources.

Gateway API model: The Kubernetes Gateway API (gateway.networking.k8s.io) is the successor to Ingress. It introduces Gateway, HTTPRoute, GRPCRoute, and TCPRoute resources with a cleaner separation of concerns between infrastructure operators (who define Gateway classes and instances) and application developers (who define HTTPRoute resources pointing to Services). Kong's implementation of the Gateway API is the preferred approach for new deployments.

The Gateway API model is better for teams because it allows platform engineers to define the GatewayClass and Gateway once, and application teams to manage their own HTTPRoute resources without cluster-level permissions.

Installing Kong on Kubernetes

# Add the Kong Helm repository
helm repo add kong https://charts.konghq.com
helm repo update

# Install Kong in Gateway API mode, DB-less, with Prometheus metrics
helm install kong kong/ingress \
  --namespace kong \
  --create-namespace \
  --set gateway.env.database=off \
  --set gateway.env.log_level=info \
  --set serviceMonitor.enabled=true \
  --set gateway.resources.requests.cpu=500m \
  --set gateway.resources.requests.memory=256Mi \
  --set gateway.resources.limits.cpu=2 \
  --set gateway.resources.limits.memory=1Gi

Gateway API Resources

# gateway-class.yaml - cluster-scoped, created by platform team
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: "true"
spec:
  controllerName: konghq.com/kic-gateway-controller

---
# gateway.yaml - cluster or namespace scoped
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong-gateway
  namespace: kong
spec:
  gatewayClassName: kong
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-cert
      allowedRoutes:
        namespaces:
          from: All

# users-httproute.yaml - created by the users team in their namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: users-route
  namespace: users
  annotations:
    konghq.com/plugins: rate-limit-users,jwt-auth
spec:
  parentRefs:
    - name: kong-gateway
      namespace: kong
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1/users
      backendRefs:
        - name: users-service
          port: 8080

---
# KongPlugin resource referenced by the HTTPRoute annotation
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limit-users
  namespace: users
plugin: rate-limiting
config:
  minute: 200
  policy: redis
  redis_host: redis.infrastructure.svc.cluster.local
  redis_port: 6379
  fault_tolerant: true

---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: jwt-auth
  namespace: users
plugin: jwt
config:
  claims_to_verify:
    - exp
  maximum_expiration: 3600

Adding Kong Mesh for East-West Traffic

With Kong Mesh, sidecar injection is controlled via namespace or pod annotations. Once the mesh control plane is installed, opt namespaces into mesh injection:

# Install Kong Mesh control plane
kumactl install control-plane | kubectl apply -f -

# Enable sidecar injection for the users namespace
kubectl label namespace users kuma.io/sidecar-injection=enabled

# mesh-traffic-policy.yaml
# Only users-service may call orders-service; deny everything else
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: orders-ingress-policy
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshService
    name: orders-service
  from:
    - targetRef:
        kind: MeshService
        name: users-service
      default:
        action: Allow
    - targetRef:
        kind: Mesh
      default:
        action: Deny

# circuit-breaker.yaml
apiVersion: kuma.io/v1alpha1
kind: CircuitBreaker
metadata:
  name: orders-circuit-breaker
  namespace: kong-mesh-system
spec:
  sources:
    - match:
        kuma.io/service: "*"
  destinations:
    - match:
        kuma.io/service: orders-service
  conf:
    interval: 5s
    baseEjectionTime: 30s
    maxEjectionPercent: 50
    detectors:
      totalErrors:
        consecutive: 5
      localOriginErrors:
        consecutive: 3

Autoscaling Kong Data Plane

Kong's stateless data plane nodes scale horizontally without coordination. Configure HPA based on CPU utilization and custom metrics from Prometheus:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kong-gateway-hpa
  namespace: kong
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kong-gateway
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: kong_http_requests_total_rate
        target:
          type: AverageValue
          averageValue: "1000"  # 1000 req/s per pod before scaling out

Plugin Ecosystem and Configuration Patterns

The Plugin Execution Model Revisited

Kong's power comes from its plugin system. There are three categories of plugins to understand:

Bundled plugins ship with Kong and cover most production needs: authentication (JWT, OAuth2, Key Auth, LDAP, OpenID Connect in enterprise), traffic control (rate limiting, request size limiting, proxy cache), transformations (request/response transformer, correlation IDs), logging (HTTP log, file log, TCP log, Kafka log), analytics (Prometheus, Datadog, StatsD), and security (IP restriction, bot detection, CORS).

Third-party plugins are installable packages distributed as Lua rocks or Docker image layers. The Kong Hub lists hundreds of community and vendor-maintained plugins.

Custom plugins are Lua modules (or, since Kong 2.x, plugins written in Go or JavaScript via Plugin Development Kit PDK) that follow Kong's schema conventions. Writing a custom plugin is the right choice when bundled plugins don't compose to meet your requirement and you need to embed business logic at the gateway layer.

Common Production Plugin Patterns

Pattern 1 - JWT + Rate Limiting per Consumer: Authenticate with JWT (which populates kong.client.get_consumer()), then rate-limit against the authenticated consumer identity rather than the source IP. This prevents rate limit bypass via proxy rotation.

plugins:
  - name: jwt
    config:
      anonymous: null  # reject unauthenticated requests entirely
  - name: rate-limiting
    config:
      minute: 1000
      policy: redis
      redis_host: redis
      limit_by: consumer  # key insight: limit by consumer, not IP

Pattern 2 - Response Caching for Read-Heavy Endpoints: The proxy-cache plugin caches upstream responses in Kong's shared memory or Redis, dramatically reducing upstream load for read-heavy APIs.

plugins:
  - name: proxy-cache
    config:
      strategy: redis
      redis:
        host: redis
        port: 6379
      response_code:
        - 200
        - 206
      request_method:
        - GET
        - HEAD
      content_type:
        - application/json
      cache_ttl: 300  # 5 minutes
      cache_control: true  # honour Cache-Control headers from upstream

Pattern 3 - OpenID Connect for Enterprise SSO: The enterprise OIDC plugin handles the full authorization code flow, token introspection, and claim mapping to Kong consumers, enabling seamless SSO integration with Okta, Auth0, Azure AD, or Keycloak without writing custom authentication code.

Avoiding Plugin Anti-Patterns

A common mistake is using the request-transformer or response-transformer plugins to encode business logic that belongs in the application. If you're writing conditional transformations based on response body content, extracting fields and constructing new payloads, or orchestrating multiple upstream calls from a plugin - stop. That belongs in a Backend-for-Frontend (BFF) service or an API composition layer, not in the gateway. Gateways should enforce policy, not implement business logic.

Another anti-pattern is applying heavy body_filter plugins globally. Body filtering is expensive because it buffers the response body in memory, prevents streaming, and runs synchronously on the Nginx worker. Apply response-transformer narrowly to specific routes where it is genuinely needed, and benchmark the latency impact under load.

Trade-offs and Pitfalls

Configuration Drift in Hybrid Teams

When multiple teams share a Kong instance, configuration drift becomes a serious operational risk. Without rigorous GitOps enforcement, production configuration diverges from the declarative files in version control. The Admin API is powerful and easy to invoke - it takes discipline to prevent ad-hoc changes that never get committed back to the repository.

The solution is to treat the Admin API as write-protected in production. All changes flow through pull requests to the declarative config files, validated by a CI pipeline that runs deck diff (Kong's declarative configuration tool) against the running instance before applying. The deck CLI makes this tractable: deck sync applies the diff between the declared config and the live state, and deck dump exports current live config for auditing.

# In CI: validate config and preview changes
deck validate --state kong-config/kong.yaml
deck diff --state kong-config/kong.yaml --kong-addr http://kong-admin:8001

# Deploy: apply the declarative config
deck sync --state kong-config/kong.yaml --kong-addr http://kong-admin:8001

The Single Point of Failure Problem

An API gateway is by definition on the critical path of every request. A misconfigured plugin, an Nginx worker crash, or a Redis connection pool exhaustion event can bring down your entire API surface simultaneously. This makes gateway configuration changes high-risk operations.

Mitigate this with: canary deployments of configuration changes (using Kong's traffic splitting or a progressive delivery tool like Argo Rollouts), separate gateway instances per environment, circuit breakers at the gateway level for upstream dependencies, and - critically - fault_tolerant: true on the rate-limiting plugin so that Redis unavailability degrades gracefully rather than blocking all traffic.

Latency Budget Overhead

Every plugin adds latency. Individually, each plugin executes in sub-millisecond time. Cumulatively, a heavy plugin chain can add 5-15ms per request at the 99th percentile. For latency-sensitive APIs where your SLO is under 50ms end-to-end, this overhead is material.

Profile your plugin chain under realistic load using Kong's built-in request latency metrics (kong_latency_ms_bucket in Prometheus). Identify which plugins contribute the most latency. Body filter plugins are the most likely culprits. If caching is causing memory pressure on Kong workers, consider offloading to Redis explicitly.

Sidecar Overhead in the Mesh

Each Envoy sidecar consumes CPU and memory - typically 50-100m CPU and 50-100Mi RAM per pod at baseline. In a cluster with 200 pods, that is 10-20 CPU cores and 10-20GB RAM consumed entirely by mesh infrastructure. At scale, this becomes a real cost concern.

Additionally, sidecar injection changes the Pod startup sequence. Init containers run before the application container, which adds startup latency. In environments with aggressive liveness probe settings, this can cause false-positive restart loops on startup. Tune initialDelaySeconds on liveness probes when running mesh-injected pods.

The Debugging Complexity Tax

When things go wrong in a system with both a gateway and a service mesh, the debugging surface multiplies. A request can fail at the gateway plugin layer, at the Kong proxy itself, at the Envoy sidecar on the client side, at the Envoy sidecar on the server side, or in the application. Correlating logs across all four layers requires consistent trace context propagation and a unified observability stack.

Establish this before you need it in an incident. Configure Jaeger or Tempo as the distributed tracing backend, set Kong's tracing_instrumentations to include request, router, and plugin spans, and configure Kuma's MeshTrace policy to propagate those contexts through Envoy. Without this, debugging east-west issues in a mesh is significantly harder than it needs to be.

Best Practices

Treat Gateway Config as Infrastructure Code

All Kong configuration - services, routes, plugins, consumers, upstreams - should live in version-controlled declarative files. Use deck for reconciliation, validate configs in CI before every merge, and enforce that the Admin API is read-only in production except via the automated deployment pipeline. This is not optional in teams larger than two people.

Structure your config repository by environment, with environment-specific values injected via deck's --values flag or via kustomize overlays for the Kubernetes CRD approach. Keep secrets (JWT secrets, API keys, Redis passwords) out of the config files and inject them from a secrets manager (HashiCorp Vault, AWS Secrets Manager) at deploy time.

Use the Gateway API on Kubernetes

For new Kubernetes deployments, use the Gateway API (HTTPRoute, GRPCRoute) rather than the legacy Ingress controller model. The Gateway API provides better separation of concerns, richer routing semantics (traffic weighting, header-based routing, query param matching), and is the direction the Kubernetes ecosystem is moving. It also makes multi-implementation scenarios (e.g., migrating from Nginx Ingress to Kong) less disruptive.

Right-Size Plugin Scope

Apply plugins at the most specific scope necessary. A JWT plugin applied globally means every route in your gateway - including internal health check endpoints and webhook receivers - requires a JWT. Instead, apply auth plugins at the service or route level, and use the anonymous consumer pattern for routes that have a mixed auth requirement (authenticated consumers get one rate limit tier, anonymous requests get a stricter one). Global plugins should be limited to truly cross-cutting concerns: correlation ID injection, global logging, and metrics collection.

Invest in Observability Before You Need It

Configure the Prometheus plugin globally from day one. Set up dashboards in Grafana for gateway throughput, error rates (4xx/5xx split), upstream latency, and plugin execution latency before you go to production. Define SLOs against those metrics. When incidents happen - and they will - the difference between a 10-minute resolution and a 2-hour investigation is having these baselines already in place.

The official Kong Grafana dashboard (available on grafana.com dashboard ID 7424) is a good starting point. Extend it with per-route and per-consumer breakdown dashboards for large deployments.

Validate Upstream Health Actively

Kong's upstream health checking feature (both active and passive) should be enabled for all production upstreams. Passive health checking detects failures from actual traffic; active health checking probes configured endpoints on a schedule regardless of traffic. Without health checking, Kong will continue routing to a crashed upstream pod until Kubernetes removes it from the Endpoints object, which can take 10-30 seconds during which requests are failing.

upstreams:
  - name: users-service-upstream
    algorithm: round-robin
    healthchecks:
      active:
        type: http
        http_path: /health
        healthy:
          interval: 5
          successes: 2
        unhealthy:
          interval: 5
          http_failures: 3
          tcp_failures: 2
      passive:
        healthy:
          successes: 5
        unhealthy:
          http_failures: 5
          tcp_failures: 2

Key Takeaways and 80/20 Insight

80/20 Insight: The Three Decisions That Matter Most

Of everything in this article, three decisions drive the majority of your operational outcomes:

1. DB-less + GitOps from day one. The choice between database-backed and DB-less Kong defines your operational model for the lifetime of the deployment. DB-less with declarative config + deck gives you auditability, reproducibility, and confidence in every change. Starting with a database and migrating later is painful. Start declarative. 2. Observe before you route. Enable Prometheus metrics and at minimum one distributed tracing backend before routing production traffic through Kong. The 30 minutes you spend configuring this will save days of debugging over the next year. 3. Gateway for north-south, mesh for east-west - don't try to do both with one tool. Kong Gateway is excellent at what it does for external traffic. Trying to route all east-west traffic through the gateway creates a bottleneck and centralizes failure risk. If east-west policy enforcement is a requirement, deploy Kong Mesh alongside Kong Gateway and let each handle its layer.

Actions You Can Apply This Week

Audit your current API surface. List every service exposed externally and identify which ones have no authentication, no rate limiting, or no observability. These are your first Kong routes.
Set up the Docker Compose PoC from this article. Experiment with plugin chaining in a local environment before touching production.
Install deck and run deck dump against your existing Kong instance (if you have one) to generate a baseline declarative config for version control.
Enable the Prometheus plugin globally on your Kong instance and wire it to Grafana. Import dashboard 7424 as a baseline.
Define one SLO for your gateway - e.g., "99th percentile proxy latency under 20ms, measured at the gateway." Alert on it. This creates an empirical baseline before you start adding plugins.

Analogies for Retention

The gateway is a customs checkpoint at a border crossing. Every person (request) entering the country (your system) passes through customs (the gateway). Customs enforces rules: valid passport (authentication), not bringing prohibited goods (request validation), quotas on how much you can import (rate limiting). Once you're inside the country, customs doesn't follow you around - that's the mesh's job (internal police, enforcing who can go where inside the country).

The service mesh is a trusted courier network. Inside your system, services communicate by handing packages (requests) to their local courier office (sidecar proxy). The courier network guarantees: only authenticated couriers can deliver packages (mTLS), every package is tracked end-to-end (tracing), and if a courier route is overloaded, packages are rerouted (circuit breaking). Neither the sender nor the receiver has to think about any of this - it's infrastructure.

Conclusion

Kong API Gateway and Kong Mesh together represent a mature, production-tested approach to managing API traffic at both the perimeter and the interior of a distributed system. Kong's strength is its composability: a small set of well-designed primitives (services, routes, plugins) that compose into sophisticated traffic management pipelines without application code changes.

The patterns presented in this article - DB-less declarative config, GitOps-managed deployments via deck, Gateway API on Kubernetes, Prometheus-based observability, and mesh-enforced east-west mTLS - are not theoretical. They reflect what production deployments at significant scale actually look like. The pitfalls (plugin latency budgets, sidecar overhead, Admin API drift, debugging complexity) are equally real and must be planned for rather than discovered in incidents.

The most important architectural principle underlying all of this: infrastructure concerns should not be solved at the application layer. Authentication, rate limiting, retries, circuit breaking, and observability belong in dedicated infrastructure that every service inherits automatically. Kong, properly configured, is that infrastructure. The goal is application code that contains only business logic, surrounded by a policy layer that handles everything else consistently, observably, and without requiring developer attention for each new service that joins the system.

References

Official Documentation

Kong Gateway Documentation (v3.x): https://docs.konghq.com/gateway/latest/
Kong Ingress Controller Documentation: https://docs.konghq.com/kubernetes-ingress-controller/latest/
Kong Mesh / Kuma Documentation: https://kuma.io/docs/latest/
decK (declarative Kong configuration): https://docs.konghq.com/deck/latest/

Kubernetes Standards

Kubernetes Gateway API specification: https://gateway-api.sigs.k8s.io/
Kubernetes Horizontal Pod Autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

CNCF and Industry References

Kuma (CNCF Service Mesh): https://kuma.io/
Envoy Proxy documentation: https://www.envoyproxy.io/docs/
OpenTelemetry specification (tracing context propagation): https://opentelemetry.io/docs/
Prometheus data model and query language: https://prometheus.io/docs/

Books and Architecture Guides

Newman, Sam. Building Microservices (2nd ed.). O'Reilly Media, 2021.
Richardson, Chris. Microservices Patterns. Manning Publications, 2019.
Burns, Brendan. Designing Distributed Systems. O'Reilly Media, 2018.

Tools Referenced

deck CLI for Kong declarative config: https://github.com/Kong/deck
Grafana Kong dashboard (ID 7424): https://grafana.com/grafana/dashboards/7424
Kong Hub (plugin ecosystem): https://docs.konghq.com/hub/
kumactl CLI: https://kuma.io/docs/latest/explore/cli/