A star in the middle of a Vault path is not a wildcard

At Gambchamp a shared HashiCorp Vault serves several client environments, and a secrets operator in one of the clusters syncs values from it into Kubernetes. One day the operator had been crash-looping for six hours — 68 restarts, every reconcile failing with:

PUT /v1/auth/token/renew-self → 403
* permission denied
* invalid token

Two problems, not one

It is easy to read only the first line and go hunting for a policy problem. The second line is the decisive one: Vault answers an invalid token with both messages. The outage was simply that the operator’s static token had expired or been revoked.

Reissuing it should have been mechanical. It was not. A fresh token carrying the environment’s three documented read-only policies could read three of the four paths the operator needs. The fourth returned permission denied, even though the policy plainly named it.

The glob that was never a glob

In Vault ACL, * is a wildcard only as the last character of a path. A + matches exactly one segment. Anywhere else, * is just a segment literally named *:

# looks like a wildcard, matches only a segment literally named "*"
path "…/frontend/*/config" { capabilities = ["read"] }

# what it had to be
path "…/frontend/+/config" { capabilities = ["read"] }

A sweep over all policies found 27 with * in the middle of a path. Vault accepts such a policy as valid — there is no linting for “this pattern can never match”.

The give-away that it was a template error rather than intent: the matching broader policy covering all clients was written correctly, with +. Which means the old token must have been carrying that broader policy all along — read access to every client’s frontend secrets, not just its own. The token expiry exposed an over-privilege that had been invisible for as long as it worked.

The fix

I did not edit 27 shared policies in the middle of an outage. Instead I created one dedicated policy with the correct + syntax, scoped to exactly the four paths the operator reads, and issued a new token with three properties that matter for an operator:

• keep default in the policy list — renew-self and lookup-self live there, and without them you recreate the same outage on a timer;
• -period instead of -ttl, so the token renews for as long as the consumer is alive;
• -orphan, so revoking the token that created it does not cascade.

Before wiring it in, I ran the exact call that had been failing (token renew), read the four required paths, and checked that another environment’s path was denied. Fixing the shared templates became a separate, unhurried task.

Takeaways

• A Vault 403 with two error lines: read the second one. invalid token is not a policy problem.
• * mid-path is a literal. Use + for one segment, and remember that Vault will not warn you.
• vault token capabilities <path> is the only thing that tells the truth about a policy.
• Never “fix” access by attaching a broader policy. It works instantly and quietly grants cross-environment reads.
• Static tokens are the real fragility. Kubernetes auth with a ServiceAccount removes this failure mode entirely.