Nginx - Advanced

Nginx as a Kubernetes Ingress Controller

Running Nginx inside Kubernetes as an Ingress controller is architecturally different from running it as a standalone reverse proxy, even though the underlying binary is the same. The ingress-nginx controller runs as a set of pods that continuously watch the Kubernetes API for Ingress resources, along with the Service and Endpoint objects they reference. When something changes, a new Ingress is created, a backend Service scales up or down, the controller regenerates a full nginx.conf from a Go template and triggers a reload, the same nginx -s reload mechanism from the previous tier, just automated and driven by cluster state instead of a human editing a file.

This has real consequences at scale. In a standalone deployment, config changes are infrequent and deliberate. In a cluster with hundreds of Ingress objects and pods constantly rolling for deploys and autoscaling, config regeneration and reload can happen very frequently, and each reload has a real (if small) cost: new workers spinning up, connection tracking being rebuilt. Clusters with heavy churn can see this become a genuine bottleneck, which is part of why the ingress-nginx project and its alternatives (and the newer Gateway API standard, meant to eventually supersede the Ingress API) have put real engineering effort into reducing reload frequency and doing more updates dynamically, without a full config regeneration, where possible.

A typical Ingress resource looks nothing like raw Nginx config, and that gap is exactly the point of the abstraction:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

Application teams write declarative YAML like this instead of hand-editing location blocks, and the controller is what turns annotations (like that proxy-read-timeout override, a direct callback to the timeout tuning covered earlier) into the literal Nginx directives underneath. This is powerful for self-service, but it also means debugging requires an extra mental hop: when something's wrong, you're not just reading nginx.conf, you're reasoning about what Ingress and Service state produced it, which is exactly why kubectl describe ingress and the controller's own generated-config dump (many ingress-nginx builds expose it via a debug endpoint or kubectl exec into the controller pod and running nginx -T there) both matter for a full picture.

---

Nginx vs a Service Mesh Sidecar

Nginx as an Ingress controller typically handles north-south traffic, requests entering the cluster from outside. Traffic between services inside the cluster, east-west traffic, is a different problem, and increasingly it's handled by a service mesh like Istio or Linkerd, built on the Envoy proxy running as a sidecar container injected into every pod.

The tradeoff is real and worth being able to argue both sides of. A sidecar mesh gives you per-service traffic policy: mutual TLS between every service without application changes, fine-grained retry and circuit-breaking behavior configured per route, gradual traffic shifting for canary deploys, all enforced consistently regardless of what language or framework each service is written in. The cost is a sidecar container running alongside every single pod in the cluster, meaning real, multiplied CPU and memory overhead, plus a meaningfully more complex system to operate and debug when something goes wrong at the mesh layer. A single central Nginx instance (or a small fleet of them) avoids that per-pod overhead entirely, but it becomes a shared chokepoint for anything routed through it, and it can't express per-service policy nearly as granularly as a mesh can. Plenty of production systems reasonably choose to use Nginx at the edge and skip a mesh entirely for internal traffic, especially when the service count and inter-service traffic complexity don't yet justify the mesh's operational cost. This is a judgment call that should track actual complexity in the system, not just what's trendy.

---

Connection Draining During Deployments

During a rolling deployment, pods behind an Nginx-backed Ingress are constantly being created and terminated, and the danger window is the moment a pod is told to terminate but Nginx hasn't yet learned that it should stop sending it traffic. Kubernetes readiness probes control when a pod starts receiving traffic in the first place, but the graceful part of removal usually relies on a preStop hook, commonly just a short sleep, that delays the container from actually shutting down until the Ingress controller has had time to notice the pod is terminating and remove it from its upstream list.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

Without that delay, there's a real race: Kubernetes can send the termination signal and start tearing the pod down before the Ingress controller's config has propagated the removal, and any in-flight requests still being routed to that pod during the gap fail with connection resets or 502s. This is a small, easy-to-miss detail that shows up as intermittent errors during deploys that are otherwise hard to reproduce, and it's a good instinct to check for whenever "we get a handful of 502s every deploy, but only sometimes" comes up as a symptom.

---