An AI agent on my ops work: the guardrails that make it safe

For the past months a large part of my operations work, in my home lab and at work, has gone through an AI agent in the terminal (Claude Code). Not a chatbot that answers questions: an agent that runs kubectl, reads logs, edits manifests, pushes to git and calls cloud APIs.

Speed is the easy part. An agent with a shell and credentials is also the fastest way to break production I have ever had access to. So the interesting question is not “how much faster”, it is “what keeps it safe”. This is the setup that answers it.

The repository is the memory

The agent starts every session with no memory. Everything it should know lives in a private git repository that it loads at the start:

• the method: how diagnosis is done, in what order, and what is never done;
• runbooks by symptom: “pod in CrashLoopBackOff”, “upgrade fails with another operation in progress”, “certificate expired behind a proxy”;
• a map of every environment: topology, workloads, network, storage, who owns what, and a dated state snapshot;
• a decision log per environment.

Knowledge accumulates in git, not in a chat window. When an incident starts, the environment is already described, and the agent reads it before it touches anything.

Rule 1: read-only first

Every step goes hypothesis → read-only check → conclusion, and only then a change. Diagnosis moves from the light tier to the heavy one: get, describe, events and logs first, then PromQL for trends and saturation, then the question that decides who fixes it: is this the application or the platform?

A symptom is not a cause. “It works sometimes” or “it fixed itself” is luck with timing, not a fix; the agent keeps looking.

Rule 2: every change shows its rollback before it runs

The agent may change things, but it never does so silently. Before a mutation I see the exact command, what it is backed up to, and the command that undoes it. Changes with a wide blast radius, such as draining a node, deleting a namespace or scaling to zero, get an explicit warning first.

My home lab has no customers and no SLA, so there the agent is allowed big-bang changes. That is the point of a lab: reconnaissance before the same change reaches an environment that needs a maintenance window and an approval.

Rule 3: short-lived, narrow credentials

Long-lived broad tokens are where an agent becomes dangerous. A real example from today: enabling bot protection on two websites at Cloudflare.

The agent did not use an account-wide key. It minted a token scoped to one zone, with only the bot-management permissions, valid for one hour, made the change, and revoked the token in a finally block. The first attempt failed: reading the setting also needed a zone-read permission. The token was revoked anyway, because revocation did not depend on success. The current state was saved before the change, together with the one-line rollback.

Secrets stay out of everything that leaves the machine: no tokens, internal hostnames or addresses in web searches, and credentials are stored encrypted in git, not pasted around.

Rule 4: the cluster does not know everything

My infrastructure is hybrid: some services run in Kubernetes, some on plain VMs. A cluster only knows about itself, so the agent is not allowed to conclude “the service is dead” from cluster data. Zero endpoints or a missing Deployment are facts about the cluster. The check is DNS against the ingress and load-balancer addresses, and then a real request to the service.

A related rule came from a near miss: every kubeconfig context has the same name, so the agent identifies a cluster by its API endpoint and nodes, not by the context.

Rule 5: decisions are written down when they are made

A significant decision gets a dated entry at the moment it is made: what was chosen, the alternatives, why, what it costs, and the trigger to revisit it. A month later the question is always “why did we pick X”, and the answer has to be in the log, not in a chat history.

Where the agent got it wrong

It does get things wrong, and the guardrails are what catch it. Three examples from a single day:

• A wrong date from a broken encoding. It read a page served in a legacy Cyrillic encoding, and the first summary reported the wrong publication month. Re-downloading and converting the page showed the real date in the structured data.
• A confident recommendation that was wrong. It told me to delete a LinkedIn skill as junk; it was the localized name of “Site Reliability Engineering”. It corrected itself once it saw the full list.
• A plan that could not work. It planned to allow an injected script in the Content Security Policy by its hash. Checking three responses showed the hash changes on every request, so the idea was dropped instead of weakening the policy.

None of these were caught because the model is clever. They were caught because every claim gets checked against the real system before anything depends on it. Trust comes from verification, not from the model.

A second opinion from different models

For risky changes and unclear incidents I use a small “council”: several models from different vendors, each with a lens (platform, application, network, a skeptic). They argue, cross-examine each other and vote until nobody blocks, and the result is a written report with the dissent kept in it. It is not a vote I obey; it is a cheap way to find the question I did not ask.

What it changes, and what it does not

The agent is a multiplier, not a replacement. A day that used to be one task is now a chain: a site release through CI and Argo CD, an analytics review, a security setting, a publication. But I still own every decision, and on production the rules for windows and approvals do not change because the hands are faster.

Takeaways

• Give the agent a written method and environment maps in git; do not rely on its memory or yours.
• Read-only first, and no change without a visible rollback.
• Mint narrow, short-lived credentials per task and revoke them in finally.
• Do not let the agent conclude anything about a service from the cluster alone.
• Log decisions when they are made.
• Expect mistakes and design for catching them: verification is the guardrail.