My homelab and client infrastructure repos all have proper CI: tfsec, Trivy, checkov, actionlint — the full stack. But there’s a tax I kept paying: write some Terraform, push, wait for the pipeline, read the findings four minutes later, context-switch back, fix it, push again. Repeat.
The feedback loop wasn’t broken — it was just slow. And slow feedback loops train you to batch-fix at the end instead of build-clean from the start.
I wanted SRE quality gates at author time — while I’m still in the file, before the commit.
The Architecture
The result is a 5-agent ecosystem built on Claude Code’s custom agent system, backed by a FastMCP 2.14.6 server.
Each agent is a single .md file in ~/.claude/agents/ with a YAML frontmatter block:
---
name: tf-auditor
description: Use this agent when you need to audit Terraform configurations...
model: claude-sonnet-4-6
---
You are a Terraform Multi-Cloud Security Auditor...Claude Code discovers these automatically at startup — no registration, no plugins, no code. The description field is what Claude uses to decide which agent to route to when you type something like “audit HomeLab terraform.”
| Agent | What it does | Backend |
|---|---|---|
tf-auditor |
tfsec + checkov + terraform validate | FastMCP tf_audit() |
obs-curator |
Prometheus rule coverage + Grafana dashboard gaps | FastMCP obs_check() + grafana_coverage() |
k8s-reviewer |
Manifest security posture (resource limits, RBAC, NetworkPolicy) | Read + Glob + kubectl dry-run |
cicd-auditor |
GitHub Actions workflow linting with actionlint | Read + Glob + Bash |
py-reviewer |
bandit + pylint + boto3 hygiene (pagination, error handling, IAM) | Read + Grep + Bash |
Two agents are MCP-backed. Three operate entirely through Claude’s built-in file tools and shell access. That split was intentional: k8s-reviewer, cicd-auditor, and py-reviewer do static analysis that reads files and shells out to CLI tools — there’s no reason to add an MCP server in the middle. tf-auditor and obs-curator needed server-side logic (subprocess orchestration, secrets masking, registry lookups, HTTP calls to Prometheus/Grafana) that’s cleaner in Python than in a shell prompt.
The MCP Server: FastMCP Makes Registration Trivial
The MCP server lives at ~/.claude/mcp-servers/sre_tools/server.py. The entire server entrypoint is 15 lines:
from fastmcp import FastMCP
from providers.terraform import tf_audit
from providers.prometheus import grafana_coverage, obs_check
mcp = FastMCP("sre-tools")
mcp.tool()(tf_audit)
mcp.tool()(obs_check)
mcp.tool()(grafana_coverage)
if __name__ == "__main__":
mcp.run()tf_audit, obs_check, and grafana_coverage are plain Python functions. mcp.tool() wraps them and exposes them over the MCP stdio protocol. This matters for testing: there’s no MCP machinery involved in the test suite at all. You just import the function and call it.
Registration is one command:
claude mcp add sre-tools \
--scope user \
/path/to/.venv/bin/python3.12 /path/to/server.pyAfter that, all five agents can call tf_audit() as if it were a local function.
The Shared Context Problem: The Project Registry
Running tfsec on “HomeLab” means knowing where HomeLab’s Terraform directory lives, which K8s context to use, which Ansible inventory file. That context can’t live in every agent — it’d go stale in six places at once.
The solution is a single project_registry.json at the MCP server root:
{
"projects": [
{
"name": "HomeLab",
"path": "/home/user/projects/homelab",
"providers": ["proxmox"],
"tf_dir": "terraform",
"ansible_inventory": "ansible/inventories/proxmox.ini",
"k8s_context": "homelab",
"ci": "github-actions"
},
{
"name": "container-observability-lab",
"path": "/home/user/projects/container-observability-lab",
"providers": ["local"],
"tf_dir": null,
"nomad": true,
"consul": true
}
],
"observability": {
"prometheus_url": "http://192.168.1.100:9090",
"grafana_url": "http://192.168.1.100:3000"
}
}Every agent reads from this one file. tf_dir: null for container-observability-lab is load-bearing — that project uses Nomad/Consul instead of Terraform. Without a null guard, tf_audit() would try to os.path.join(project_path, None) and blow up with an unhelpful TypeError. The guard catches it early:
if project["tf_dir"] is None:
return {"error": f"{project_name} has no Terraform directory (tf_dir: null)"}The K8s Reviewer: A Pure Prompt Agent
The k8s-reviewer has no Python backend. The entire agent is a structured prompt that tells Claude to use Glob to find manifests, Read to inspect them, and apply a security checklist:
## Security Checklist (apply to every Deployment, StatefulSet, DaemonSet, Pod)
1. Resource limits — resources.requests AND resources.limits must be set
2. Read-only root filesystem — securityContext.readOnlyRootFilesystem: true
3. Non-root user — securityContext.runAsNonRoot: true
4. No privileged escalation — securityContext.allowPrivilegeEscalation: false
5. No host network/PID — hostNetwork: true must be justified in a comment
6. NetworkPolicy exists — every namespace with a Deployment needs at least one
7. No :latest image tags — tags must be pinned
8. RBAC least privilege — no ClusterRoleBinding to system:unauthenticated
9. PodDisruptionBudget — prod Deployments with >1 replica should have a PDBThe output format is prescribed in the prompt too — a findings table with file:line references and remediation snippets. This means the LLM isn’t inventing structure; it’s filling in a template.
The insight here is that for static analysis of text files, you don’t need a backend at all. The model can read YAML as well as any parser for the purposes of flagging allowPrivilegeEscalation: true or a :latest tag.
The Implementation Gotchas
This is where the real engineering happened.
Secrets masking order matters — and it’s subtle.
tf_audit() shells out to terraform show -json and pipes the output through a masking function before it touches logs or gets returned to Claude. The original implementation called _mask_tfvars_secrets() on the parsed dict after json.loads(). That’s wrong — the raw subprocess stdout exists as an unmasked Python string for a brief window before parsing. The fix was to mask the raw bytes first:
raw_stdout = proc.stdout # raw string from subprocess
masked_stdout = _mask_tfvars_secrets(raw_stdout)
parsed = json.loads(masked_stdout) # parse the cleaned versionThe masking function uses a regex that covers common credential variable names — access keys, passwords, tokens, private keys. Anything matched gets replaced with ***REDACTED***. The test for this verifies both directions — that secrets are removed AND that non-secret values are preserved:
def test_mask_tfvars_secrets_redacts_values():
raw = 'access_key = "AKIA...EXAMPLE"\nregion = "us-east-1"'
masked = _mask_tfvars_secrets(raw)
assert "AKIA...EXAMPLE" not in masked
assert "us-east-1" in masked # non-secret values preserved
assert "***REDACTED***" in maskedSeverity filtering should fail loudly.
Early versions of _severity_passes() had a silent fallback: unknown severity thresholds would evaluate as True (pass everything through). That’s dangerous — a fat-fingered config like "HIHG" would suppress no findings instead of crashing. Changed it to raise ValueError:
SEVERITY_ORDER = ["LOW", "MEDIUM", "HIGH", "CRITICAL"]
def _severity_passes(finding_severity: str, threshold: str) -> bool:
if threshold not in SEVERITY_ORDER:
raise ValueError(
f"Unknown severity threshold: {threshold!r}. Valid: {SEVERITY_ORDER}"
)
return SEVERITY_ORDER.index(finding_severity) >= SEVERITY_ORDER.index(threshold)A loud failure beats a silent wrong answer every time.
Grafana lies about its HTTP status codes.
If your Grafana API token is invalid or expired, the search endpoint returns HTTP 200 with a body of {"message": "unauthorized"}. If you check response.status_code == 200 and then iterate response.json() as a list, you get a KeyError or TypeError — not the 401 you’d expect.
The fix is a type guard before iteration:
dashboards = response.json()
if not isinstance(dashboards, list):
return {
"total_dashboards": 0,
"dashboards": [],
"error": f"Unexpected response type from Grafana: {type(dashboards).__name__} — check API token"
}This is also tested explicitly:
def test_grafana_coverage_handles_non_list_response():
mock_resp = MagicMock()
mock_resp.json.return_value = {"message": "unauthorized"}
with patch("providers.prometheus.httpx.get", return_value=mock_resp):
result = grafana_coverage(folder=None)
assert result["total_dashboards"] == 0
assert "unexpected response type" in result["error"].lower()pyenv breaks venv symlinks.
MCP server registration in Claude Code requires an absolute path to the Python binary. My .venv/bin/python symlink pointed to pyenv’s shim — not the actual Python 3.12 interpreter. When pyenv’s shim runs, it resolves the active Python version from .python-version files, but in the context of a background MCP process (no working directory, no shell profile loaded), it can resolve to the wrong version or fail entirely.
The fix: always register with the explicit versioned binary:
# Fragile — resolves through pyenv shim
claude mcp add sre-tools --scope user .venv/bin/python server.py
# Stable — bypasses pyenv entirely
claude mcp add sre-tools --scope user .venv/bin/python3.12 server.pyThis is the kind of thing that works fine in interactive testing (where your shell profile sets up pyenv correctly) and fails silently in background processes. Running the tests from a fresh terminal caught it.
pytest path isolation across invocation directories.
The MCP server lives at ~/.claude/mcp-servers/sre_tools/. The test suite imports from providers.terraform import tf_audit. When pytest is run from the repo root, providers/ isn’t on sys.path — it’s three directories deep.
The standard fix is a conftest.py at the package root that inserts the directory at import time:
# sre_tools/conftest.py
import pathlib, sys
sys.path.insert(0, str(pathlib.Path(__file__).parent))pytest loads conftest.py files before collecting tests, so this runs regardless of invocation directory. The full 16-test suite now passes whether you run pytest from the repo root, from sre_tools/, or from anywhere else.
Testing Strategy: Mock the Boundary, Not the Logic
The test suite has 16 tests across test_terraform.py and test_prometheus.py. The design principle: mock at the I/O boundary (subprocess calls, HTTP requests), never inside the logic.
For example, test_tf_audit_calls_checkov_and_tfsec patches subprocess.run to return pre-canned JSON — it never actually runs tfsec or checkov. But _parse_checkov_output and _parse_tfsec_output are tested against real fixture data with no mocking at all. This means the parsing logic is tested in isolation and the orchestration is tested with controlled I/O.
def test_tf_audit_calls_checkov_and_tfsec(tmp_path):
mock_checkov = MagicMock()
mock_checkov.returncode = 0
mock_checkov.stdout = json.dumps(CHECKOV_JSON)
mock_tfsec = MagicMock()
mock_tfsec.returncode = 0
mock_tfsec.stdout = json.dumps(TFSEC_JSON)
fake_project = {
"name": "test-proj",
"path": str(tmp_path),
"tf_dir": ".",
"providers": ["aws"]
}
with patch("providers.terraform._REGISTRY", {"projects": [fake_project]}):
with patch("providers.terraform.subprocess.run") as mock_run:
mock_run.side_effect = [mock_checkov, mock_tfsec]
result = tf_audit("test-proj", checks=["checkov", "tfsec"], severity_threshold="MEDIUM")
assert result["summary"]["total_high"] == 2 # 1 from checkov + 1 from tfsecThe _REGISTRY patch is worth noting: rather than creating real project directories on disk, we inject a fake registry entry pointing to tmp_path. This keeps tests hermetic and fast.
The Result
Sixteen tests passing. Five agents, each invokable with natural language from any directory. The feedback loop that used to cost a push-wait-read-fix-push cycle now costs a 30-second in-editor check.
The agents are deliberately stateless. They don’t learn from previous runs, they don’t coordinate, they don’t store findings across sessions. Each one does one job on demand. The intelligence is in the prompts, the shared registry, and the Python logic — not in runtime orchestration.
What didn’t go as planned: I underestimated how much the “test it exactly like it runs in production” principle would surface environment-specific bugs. The pyenv symlink issue, the masking order issue, and the Grafana 200-with-error-body issue all came out of writing tests that mimicked real subprocess boundaries rather than mocking too high up the stack.
What’s Next
The obvious extension is a pre-commit hook that auto-invokes the relevant agent based on changed files: Terraform files changed → tf-auditor, workflow YAML changed → cicd-auditor, Python Lambda changed → py-reviewer. Zero manual invocation required.
CI is still there for the final gate. These agents just make sure the obvious stuff is caught before it gets that far — and before you’ve lost the mental context of what you were building.