A Postgres replica stuck in a pg_rewind loop for 23 hours: rebuild, do not repair

In my home lab a side project ran on a two-instance CloudNativePG cluster: Postgres 16 on Longhorn storage, WAL archived to object storage. One day the cluster reported “2 instances, 1 ready”. The replica’s readiness probe had been failing with HTTP 500 for 23 hours — almost nine thousand failures. The primary was healthy the whole time, so the application never went down. Only high availability was gone.

What pg_rewind said

After a failover the timeline moved from 14 to 15, and the demoted instance had to rewind onto the new timeline. It could not:

pg_rewind: servers diverged at WAL location 23/4B000000 on timeline 14
pg_rewind: error: could not read file ".../pg_wal/0000000E000000230000004A": read 0 of 8192

The replica’s local WAL segments around the divergence point were zero bytes, all stamped at the moment of the incident. Its data directory was 407 MB against 4.5 GB on the primary. With no WAL to read, rewind exits with an error, the instance manager retries, and readiness never turns green. Forever.

It was not the disk

My first suspect was storage: there is a degrading SSD under that host. But the Longhorn volume was attached and healthy, both volume replicas were running, nothing had failed. The storage was innocent this time.

There was nothing to repair

The data directory was not stale — it was gone. So the fix was to throw the replica away and let the operator seed a new one from the primary:

kubectl -n <ns> delete pvc <replica> --wait=false
kubectl -n <ns> delete pod <replica>

The operator created a fresh volume, ran a join pod with pg_basebackup, started the instance, reached a consistent recovery state and began streaming. About four minutes for 4.5 GB. The primary then showed the replica streaming with zero lag.

The check that almost fooled me

To confirm the replica had the data I looked at pg_stat_user_tables.n_live_tup. It said zero rows in total. For a second it looked like an empty database.

It was not. That view holds local planner statistics. Autovacuum and analyze do not run on a standby, and statistics are not replicated, so it is legitimately zero on a perfectly healthy replica. The real check is to run the same count(*) on both sides and compare pg_current_wal_lsn() on the primary with pg_last_wal_replay_lsn() on the replica. Both databases matched to the row.

Why nobody noticed for 23 hours

• The operator’s metrics were not scraped at all: the cluster had enablePodMonitor: false.
• Enabling it was not enough. kube-prometheus-stack only picks up PodMonitors with its own release label, and CloudNativePG labels them with its own labels only. The PodMonitor was created and silently ignored until I added the label through podMonitorMetadata.

Takeaways

• A replica that cannot find its WAL is not a repair job. Rebuild it from the primary; the operator does it in minutes.
• Do not judge a standby by n_live_tup. Compare real counts and LSNs.
• Check that your monitoring stack actually selects the PodMonitor you created. “Created” and “scraped” are different things.
• Losing high availability silently is an incident even when users see nothing.