From ingress-nginx to Envoy Gateway with zero downtime: 27 Ingresses, 39 HTTPRoutes

At Optimacros a Kubernetes cluster serves the company’s public marketing sites. Community ingress-nginx was retired in March 2026, and its last releases cap the Kubernetes version: one line tops out at 1.33, the final one at 1.35. Upgrading to 1.36 without replacing the ingress was impossible.

The result: 16 hosts moved to Gateway API, 27 Ingress objects became 39 HTTPRoutes, zero differences from the old behaviour, and production never went down.

The scheme that made it safe

The key was not the manifests but the order. The new entry point comes up on a free address next to the old one, and only the address is swapped at the very end.

1. Envoy Gateway on a spare address while production stays on ingress-nginx.
2. Compare all 16 hosts on both addresses.
3. Swap the two addresses with a single annotation each. This is the only moment with any impact.
4. The old ingress stays alive on the spare address, so rollback is one annotation.

The comparison was not “the page opens”. For every host I compared the response code and the redirect code through the Host header and SNI on both addresses at once. 16 of 16 matched.

Trap 1: the cluster domain is not always cluster.local

This cluster inherited a non-default cluster domain from kubespray. With the chart default, the proxy pod resolved its xDS server to a name that does not exist, got NXDOMAIN and sat at 1/2 Running with no healthy upstream. cat /etc/resolv.conf in any pod shows the real search domain. The same domain goes into the SAN of the certificates the chart generates, so changing the value is not enough: the certificates must be reissued too.

Trap 2: a Helm hook does not rerun on refresh

The cert generator is a pre-install and pre-upgrade hook. If the Application is already Synced, Argo CD sees nothing to do and never runs it, and the hook’s ServiceAccount is deleted after the run as well. I rendered that set with helm template, stripped the hook annotations and applied it by hand.

Trap 3: basic auth only in {SHA} format

ingress-nginx stored apr1 hashes. Envoy Gateway’s SecurityPolicy rejects them with unsupported htpasswd format: please use {SHA}, and the key must be named .htpasswd. Where the password was known, I recomputed the hash. Where only an apr1 hash existed, the password could not be recovered, only rotated. I kept the old key next to the new one: while both formats sit in the same Secret, both ingresses work and rollback stays alive.

Trap 4: annotations without a direct equivalent

ingress-nginx	Gateway API
ssl-redirect	a separate HTTPRoute on the :80 listener with RequestRedirect, status 308 like nginx
auth-type: basic	SecurityPolicy.basicAuth on a separate route, or the whole site asks for a password
rewrite-target with regex	often just a prefix swap: ReplacePrefixMatch
backend-protocol: HTTPS	BackendTLSPolicy, which needs a stable CA
named backend port	numbers only in backendRefs.port

Argo CD’s server generates a fresh in-memory certificate on every start, so there is no CA to build a BackendTLSPolicy from. Behind external TLS termination the standard answer is server.insecure=true, together with removing the HTTPS backend annotation from the old Ingress at the same moment.

Trap 5: the inventory lies until you check the network

• Eight TLS secrets turned out to be two certificates: one wildcard stored five times under different names, already covering the apex.
• Two referenced secrets did not exist at all. Those hosts had served the ingress controller’s fake certificate for years, and nobody had noticed.
• Two Ingresses were leftovers of a deleted application: no DNS, no Service, 503 in production. They were deleted, not migrated.

Small things that ate time

• A kustomize directory applies only what is listed in resources:. A file placed next to it is silently ignored.
• A Kyverno policy that cloned a TLS secret when an Ingress appeared does not fire for a Gateway. It needs its own rule and RBAC for gateways.
• App-of-apps refreshes top-down: refreshing a child while its parent is on an old revision does nothing.
• A cross-namespace Secret reference from a Gateway needs a ReferenceGrant in the owner’s namespace.

Takeaways

• Run the new entry point beside the old one and swap only the address. That makes cutover and rollback one annotation each.
• Compare status and redirect codes host by host on both addresses before switching.
• Check the cluster domain before installing anything that resolves in-cluster names.
• Inventory TLS secrets against what production actually serves.
• Do not delete the old ingress on cutover day. It is your rollback.