← All posts

One edit to a LoadBalancer Service, and prod resets every TLS connection

On EKS with nginx-ingress behind an NLB, changing the Service recreates the target groups and silently drops proxy_protocol_v2. No status code, healthy targets, nothing in Git — and a quiet environment hides it for days.

The symptom was the least helpful kind: curl against the public hostname failed with OpenSSL SSL_connect: Connection reset by peer. No HTTP status at all. The NLB targets were healthy, the ingress controller pods were Running with zero restarts, Argo CD was green.

The controller log had one useful line, repeated: broken header: "" while reading PROXY protocol.

What happened

The setup is common: nginx-ingress on EKS, exposed through a Network Load Balancer created by the in-tree cloud provider from a Service of type LoadBalancer. To keep the real client IP, the target groups have proxy_protocol_v2.enabled=true, and nginx listens with use-proxy-protocol: true.

A few days earlier someone had edited that Service. When the Service changes — externalTrafficPolicy, annotations, ports, selector — the in-tree provider recreates the target groups. The new groups come up with proxy_protocol_v2.enabled=false. The attribute is not carried over.

nginx still waits for a PROXY header. It receives a bare TLS ClientHello, tries to parse it as a header, fails and closes the connection. Hence the reset with no status code.

The important part: this attribute is not in Git and cannot be. It lives only in AWS. Helm does not know about it, Argo CD does not know about it, no sync will ever bring it back.

How to see it

Check the attribute on every target group in the account and match them by NodePort:

for arn in $(aws elbv2 describe-target-groups --query 'TargetGroups[].TargetGroupArn' --output text); do
  pp=$(aws elbv2 describe-target-group-attributes --target-group-arn "$arn" \
        --query 'Attributes[?Key==`proxy_protocol_v2.enabled`].Value' --output text)
  port=$(aws elbv2 describe-target-groups --target-group-arns "$arn" \
          --query 'TargetGroups[0].Port' --output text)
  echo "$(basename "$arn") port=$port proxy_v2=$pp"
done

If several clusters share one account, match target groups by NodePort and by the address behind the NLB DNS name, never by name. Auto-generated names like k8s-ingress-ingress-* look identical, and a fix can easily land on the wrong environment.

The fix

aws elbv2 modify-target-group-attributes --target-group-arn <arn> \
  --attributes Key=proxy_protocol_v2.enabled,Value=true

Apply it to all target groups of both load balancers, public and internal. Targets turn healthy in about 30 seconds. Then verify with any status code instead of a reset: a 400 or 403 means alive, a reset means not yet.

The mine that triggers it

The edit itself was an attempt to switch the Service to externalTrafficPolicy: Local. That does not work with PROXY protocol: Local moves the health check to kube-proxy’s healthz port, which answers plain HTTP, while the target group expects a PROXY header. All targets go unhealthy and the whole ingress goes down.

Rolling back to Cluster recreates the target groups a second time, and they lose the attribute again. One bad edit costs two manual repairs. And it is not even needed: with Cluster, the client IP is already preserved by the PROXY protocol itself.

Why logs and monitoring missed it

Two reasons, both worth knowing.

The controller writes a JSON access log at high volume and the kubelet rotates the file within minutes. kubectl logs | grep for a window full of errors returns zero, which reads as “no errors ever”. Measure a live window instead:

timeout 100 kubectl -n ingress logs -f <controller-pod> --tail=0 | grep -c 'broken header'

And on an environment without traffic, nothing complains at all: monitoring is quiet, pods are healthy, targets are healthy. A production environment stayed like this for several days before a manual check found it.

Takeaways

  • Any change to a LoadBalancer Service on the in-tree provider may recreate target groups. Treat it as a change to AWS, not to Kubernetes.
  • State that lives only in the cloud is invisible to GitOps. Write it down and check it explicitly after every Service edit.
  • externalTrafficPolicy: Local and PROXY protocol on an NLB do not mix. You do not need Local to keep the client IP.
  • “Connection reset, no status code” plus broken header in nginx means a PROXY protocol mismatch, not TLS.
  • Count errors in a live window when logs rotate faster than you can read them.