Ingress and ingress controllers - deep dive

Ingress is the seam between the public internet and your cluster. Get it wrong and you get DDoS-able edges, certificate fire drills, and 502s on every deploy. Get it right and the rest of the cluster is internal and quiet.

What Ingress actually does

The Ingress resource is a routing table:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.example.com]
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service: { name: users-svc, port: { number: 80 } }
          - path: /orders
            pathType: Prefix
            backend:
              service: { name: orders-svc, port: { number: 80 } }

That object does nothing on its own. An Ingress controller watches Ingress resources matching its ingressClassName and translates them into real routing config.

Two architectures

In-cluster proxy (ingress-nginx, Traefik, HAProxy):

A Deployment of proxy pods. A Service of type LoadBalancer fronts them. Cloud LB sends traffic to proxy pods, proxy pods route to backend Services. Two hops, but proxy gives you a lot of features (rewrites, rate limits, snippet config, modsec WAF).

Cloud-native (AWS ALB Controller, GCP GKE Ingress):

Controller watches Ingress objects and provisions actual cloud LBs (ALB on AWS). Target type ip registers Pod IPs directly as ALB targets. No in-cluster proxy, no extra hop. The cloud LB does the L7 routing. You get all the cloud LB features (WAF, ACM certs, access logs to S3).

I have run both. For EKS, AWS Load Balancer Controller wins for public-facing traffic. ingress-nginx is fine for internal admin UIs or where you need NGINX-specific features.

ingress-nginx specifics

The pod runs NGINX plus a controller process that watches the API server and writes nginx.conf. Most changes use nginx -s reload (graceful), some require restart. Worth knowing:

• Default LB Service type pins external IPs on cloud. You usually want a single LB per cluster fronting nginx, not one per Ingress.
• nginx.ingress.kubernetes.io/proxy-body-size: 50m is the upload size knob. Default 1m bites you the moment someone POSTs a file.
• Rate limits like nginx.ingress.kubernetes.io/limit-rps: "10" are per-replica. With 4 NGINX pods your cluster-wide RPS limit is 40.
• use-proxy-protocol: "true" is mandatory if you front it with an NLB and need real client IPs. Both sides must agree.
• Snippets (custom Lua/NGINX config) are disabled by default since CVE-2021-25742. Enable carefully.

AWS Load Balancer Controller specifics

It is a controller that creates AWS ALB or NLB resources from Kubernetes Ingress (ALB) or Service of type LoadBalancer (NLB).

• alb.ingress.kubernetes.io/target-type: ip is the line that matters. Without it, ALB targets are NodePorts (instance mode), with it targets are Pod IPs registered directly with the ALB. Instance mode means an extra hop and SNAT. IP mode is the cloud-native path.
• One Ingress = one ALB by default. Use alb.ingress.kubernetes.io/group.name: shared to consolidate multiple Ingresses behind a single ALB. Saves $20/month per consolidated ALB. We used this at Binocs.
• ACM cert ARNs via alb.ingress.kubernetes.io/certificate-arn. No cert-manager needed for public certs - just point at ACM.
• WAF, Shield, access logs to S3, all configurable via annotations.

TLS and cert-manager

For non-ACM certs (private CAs, Let's Encrypt for internal hostnames) cert-manager is the standard. Install it, create an Issuer or ClusterIssuer, add cert-manager.io/cluster-issuer: letsencrypt-prod to your Ingress, and cert-manager handles the ACME flow:

1. Creates a Certificate resource.
2. Creates an Order, which creates Challenges (HTTP-01 or DNS-01).
3. For HTTP-01: temporarily injects an Ingress rule that serves the challenge token.
4. Let's Encrypt verifies, issues, cert-manager stores in a Secret.
5. Renewal happens automatically at 2/3 lifetime.

DNS-01 challenges (via Route53, Cloudflare) are required for wildcard certs and work better behind WAFs that interfere with HTTP-01.

Gateway API: the replacement for Ingress

Ingress was designed in 2016 and has not aged well. Gateway API is the replacement, went GA in 1.29.

Three resources, three personas:

• GatewayClass: who is providing the implementation (NGINX, Envoy, Cilium, Istio).
• Gateway: the actual listener: hostnames, ports, TLS, allowedRoutes.
• HTTPRoute / GRPCRoute / TCPRoute / TLSRoute: routing rules, owned by app teams in their own namespaces.

Why it is better:

• Multi-tenancy: the Gateway can allow specific routes from specific namespaces.
• Typed protocol support instead of cramming everything into HTTP Ingress + annotations.
• Cross-namespace references with explicit ReferenceGrant rather than implicit annotation hacks.
• Standard support for header rewriting, request mirroring, traffic splitting (canaries).

For new clusters, start with Gateway API. For existing ingress-nginx clusters, migrate when you next replace your ingress controller.

Operating ingress in production

Things that bit me at Binocs (and that you should preempt):

1. Single point of failure: if you have 1 ingress controller pod, a node drain takes your edge down. Run at least 3, spread across zones, PDB with minAvailable: 2.
2. Slow config reloads: ingress-nginx reload time scales with the number of Ingress objects. At 200+ Ingresses you start seeing tens of seconds per reload. Consider per-team controllers with different ingressClassName.
3. 502 storms on rollouts: backend Pod terminates before ingress controller refreshes endpoints. Use preStop sleep on backends AND proxy-next-upstream to retry on closed connections.
4. Cert expiry alerts: even with cert-manager, monitor cert expiry separately. The ACME flow can fail silently. We had Prometheus alerts firing at T-14 days remaining.
5. WAF in front: if you are exposing to the public internet without Cloudflare or AWS WAF in front, you are one CVE away from a bad week.

Cost angle

One ALB is roughly $20/month plus LCU usage. Pre-consolidation we had 12 Ingresses each provisioning its own ALB - $240/month just for the LBs. Grouping them with alb.ingress.kubernetes.io/group.name collapsed it to 2 ALBs. That $200/month was a chunk of the broader EKS right-sizing work that eventually compounded into the $1.8k-$2k/month savings.

The interview narrative

Open with the split: Ingress is config, the controller is the worker, you pick the controller per cluster. Mention ingress-nginx as the default choice, AWS Load Balancer Controller for EKS production with target-type ip. Talk about cert-manager for TLS automation. Close with Gateway API as the strategic direction for new clusters and why (multi-tenancy, typed routes, cleaner separation of concerns).