Building Local AI DevOps Agents with Hermes and Clanker Cloud

Most AI DevOps tools have a quiet problem: they're cloud-hosted. Every natural-language query about your infrastructure, every generated deployment plan, every incident triage prompt — it all leaves your network and lands on someone else's servers. For personal side projects, that tradeoff is fine. For regulated industries, government contracts, or organizations with strict data handling policies, it's a blocker.

This guide covers a mode that reduces model-data egress: Hermes 3 running locally via Ollama, wired into Clanker Cloud's local MCP server. Model prompts can remain local and raw provider credentials stay in the desktop credential chain, but provider API calls and ordinary Clanker Cloud account, security, download, or update traffic still use network services. Optional hosted features have separate data paths.

---

Why Local Models Matter for DevOps Agents

When you describe infrastructure to a cloud-hosted LLM, selected operational data leaves the local environment. Whether that is permitted in healthcare, finance, or government depends on the actual data, agreements, provider approvals, control scope, and system authorization—not the industry label alone.

Beyond compliance, there are practical reasons to prefer local inference for DevOps agents:

• No per-token cost. A cost audit agent that runs on a daily cron schedule against your full cloud inventory would cost real money if routed through OpenAI or Anthropic. Running locally, it costs electricity.
• Offline operation. Incident response shouldn't depend on whether api.openai.com is reachable. A local agent works even when external services are degraded.
• No rate limits. Cloud APIs throttle. A local model doesn't.
• Auditability. You control every inference call. Nothing is logged by a third party.

The missing piece, historically, was that local models weren't reliable enough at structured function calling to be useful in agent frameworks. That has changed.

---

What Hermes 3 Is and Why It Works Here

Hermes 3 is an open-source model series from NousResearch, fine-tuned on Meta's Llama 3.1 base (available in 8B, 70B, and 405B parameter sizes). Where most fine-tunes optimize for general conversational quality, Hermes 3 was explicitly trained for:

• Reliable function calling. Hermes uses a structured <tool_call> / <tool_response> format within the ChatML prompt template. The model consistently outputs parseable JSON function calls without needing special prompting tricks.
• Agentic multi-step reasoning. Hermes 3 supports a <scratch_pad> mechanism — a Goal-Oriented Action Planning (GOAP) framework where the model reasons through a task, plans its tool calls, observes results, and reflects before taking the next step. This is critical for non-trivial DevOps workflows.
• Structured output / JSON mode. When you need the agent to produce a typed report (cost anomalies, config drift, triage summary), Hermes can output strict JSON conforming to a Pydantic schema.

The Hermes 3 technical report describes it directly: "When combined, Hermes 3 can perform planning, incorporate outside data, and make use of external tools in an interpretable and transparent manner out-of-the-box, making it an excellent choice for agentic tasks."

For DevOps agent work specifically, the 8B variant handles most tasks with low latency on a machine with 16GB of RAM. The 70B variant produces noticeably more careful reasoning for complex multi-step problems like incident triage across distributed systems.

---

The Stack: Hermes + Ollama + Agent Framework + Clanker Cloud MCP

Here is how the four layers fit together:

┌─────────────────────────────────────┐
│         Agent Framework             │  CrewAI / LangChain / AutoGen
│  (task decomposition, tool routing) │
└────────────────┬────────────────────┘
                 │
┌────────────────▼────────────────────┐
│           Hermes 3 via Ollama       │  Local inference, function calling
│       (LLM brain of the agent)      │
└────────────────┬────────────────────┘
                 │ MCP protocol
┌────────────────▼────────────────────┐
│        Clanker Cloud Desktop        │  Live infra context, auth layer
│   (MCP server + cloud connectors)   │
└────────────────┬────────────────────┘
                 │
┌────────────────▼────────────────────┐
│    Cloud Providers & Platforms      │  AWS · GCP · Azure · K8s · GitHub
│                                     │  Cloudflare · Hetzner · DigitalOcean
└─────────────────────────────────────┘

Hermes 3 is the language model. It receives tool schemas from the agent framework, decides which tools to call, parses the responses, and generates plans or reports.

Ollama is the local inference runtime. It serves Hermes via an OpenAI-compatible API endpoint (http://localhost:11434), which means any LangChain or CrewAI code that can talk to OpenAI can talk to Hermes locally with a one-line change.

The agent framework (CrewAI, LangChain, or others) handles task decomposition, agent roles, memory, and tool invocation. It is the orchestration layer that turns Hermes's function calling into multi-step workflows.

Clanker Cloud is the infrastructure layer. In the normal desktop workflow it holds raw cloud credentials locally and exposes connected infrastructure through an authenticated local MCP server. The agent framework need not receive those raw credentials, although selected tool results are provided to the local Hermes model and any other enabled routes must be assessed separately.

Clanker Cloud also operates read-first. Before any change is applied, it generates a reviewed plan. Changes only execute in explicit "maker mode." This means a Hermes agent querying infrastructure context through Clanker Cloud has safe read access by default — it cannot accidentally delete a production database.

---

Setup Walkthrough

Step 1: Install Ollama and pull Hermes 3

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start the Ollama server
ollama serve

# Pull Hermes 3 (8B is a good starting point)
ollama pull nous-hermes-3

# Verify it runs
ollama run nous-hermes-3 "List three Kubernetes health checks in JSON"

The model name on Ollama is nous-hermes-3 (also available as hermes3). It maps to the NousResearch Hermes-3-Llama-3.1 series.

Step 2: Install Clanker Cloud desktop

Download the desktop app and connect your cloud providers. In the normal desktop provider workflow, raw provider credentials stay in the local credential chain and are not sent to Clanker Cloud. This does not mean account metadata or content submitted to hosted features remains local.

Clanker Cloud is a BYOK (bring your own keys) platform. Hermes via Ollama is one of the supported local model options alongside Gemma 4, Claude Code, and Codex — no external API key required for local inference.

Once your providers are connected, the MCP server endpoint is available at http://localhost:PORT/mcp (the exact port is shown in the desktop app settings). See the full docs at docs.clankercloud.ai for the MCP server configuration.

Step 3: Install your agent framework

pip install crewai langchain-community langchain

Step 4: Wire Hermes to Clanker Cloud MCP

LangChain example:

from langchain_community.llms import Ollama
from langchain.agents import AgentExecutor, create_react_agent
from langchain_mcp_adapters.client import MultiServerMCPClient

# Local Hermes model — no cloud-model key; model prompt stays local
llm = Ollama(model="nous-hermes-3", base_url="http://localhost:11434")

# Connect to Clanker Cloud's MCP server for live infra tools
async def build_agent():
    async with MultiServerMCPClient(
        {
            "clanker-cloud": {
                "url": "http://localhost:3100/mcp",  # Port from Clanker Cloud desktop
                "transport": "streamable_http",
            }
        }
    ) as client:
        tools = client.get_tools()
        agent = create_react_agent(llm, tools)
        executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
        return executor

CrewAI example:

from crewai import Agent, Task, Crew, LLM

# Hermes 3 via Ollama
hermes_llm = LLM(
    model="ollama/nous-hermes-3",
    base_url="http://localhost:11434",
    temperature=0.1,  # Low temperature for consistent structured output
)

# Clanker Cloud MCP tools are loaded separately via mcp_server_url
devops_agent = Agent(
    role="Infrastructure Analyst",
    goal="Query live infrastructure state and produce triage reports",
    backstory="A senior SRE with deep knowledge of AWS, GCP, and Kubernetes.",
    llm=hermes_llm,
    verbose=True,
)

For team workflows, see the AI DevOps for teams guide for multi-agent patterns. For getting from prototype to production, the vibe coding to production guide covers the deployment path.

---

Use Case 1: Incident Triage Agent

An alert fires at 2 AM. A pod is crashlooping in your production Kubernetes cluster. Before you dig in manually, your Hermes triage agent can do the first pass.

The agent wakes (triggered via webhook or alertmanager integration), connects to Clanker Cloud's MCP server, and executes a sequence of read queries:

1. List recent deployments in the affected namespace (last 2 hours)
2. Pull current pod status and restart counts
3. Fetch recent config changes from the Clanker Cloud change log
4. Cross-reference with GitHub — any recent merges to the production branch?

Hermes's <scratch_pad> reasoning framework makes it well-suited to this sequential investigation. It plans what to check, observes results, and revises its hypothesis before writing a triage report.

Example triage task:

triage_task = Task(
    description="""
    A CrashLoopBackOff has been detected for pod {pod_name} in namespace {namespace}.
    Query Clanker Cloud for:
    1. Recent deployments in the namespace (last 2 hours)
    2. Current pod resource limits vs. actual usage
    3. Any config changes in the last 24 hours
    Generate a structured triage report with probable root cause and next steps.
    """,
    expected_output="JSON triage report with probable_cause, evidence, and recommended_actions fields",
    agent=devops_agent,
)

With local Hermes and the local MCP path, the model prompt and selected tool results can remain on the machine. Cloud-provider queries still go to the relevant provider, ordinary account or update traffic may occur, and enabling hosted inference, sandboxes, voice, or remote control changes the boundary.

---

Use Case 2: Cost Audit Agent

Cloud costs drift. A forgotten load balancer, an oversized RDS instance, a Lambda function that started hitting a paid tier — these accumulate quietly. A cost audit agent on a daily cron is a cheap way to surface anomalies before they become surprises.

# cron: 0 8 * * 1  (every Monday at 8 AM)

cost_audit_task = Task(
    description="""
    Connect to Clanker Cloud and retrieve the last 7 days of cloud cost data across
    all connected providers. Identify:
    - Any service with >20% week-over-week cost increase
    - Resources running in unexpected regions
    - Idle resources (compute with <5% average CPU over 7 days)
    Write the findings to /reports/cost-audit-{date}.json
    """,
    expected_output="Cost anomaly report with resource ID, current cost, baseline cost, and anomaly type",
    agent=devops_agent,
)

The agent handles auth by querying through Clanker Cloud — the cloud provider credentials stay in the desktop app, not in the agent code. No AWS access key is ever passed to Hermes.

---

Use Case 3: Pre-Deploy Config Review

Before pushing a deployment, it is useful to know whether the target environment matches what you're expecting. Config drift is a common source of failed deploys — a Kubernetes namespace that was manually patched, an environment variable that was never updated, a service account that was rotated without updating the secret.

A Hermes agent connected to Clanker Cloud can automate this check as a CI/CD step:

pre_deploy_task = Task(
    description="""
    Before deploying {service_name} version {version} to production:
    1. Query the current production Kubernetes config for {service_name} via Clanker Cloud
    2. Compare actual config against the expected config in {config_file}
    3. Flag any mismatches: image tags, replica counts, resource limits, env vars, secrets
    4. Return PASS or FAIL with a diff report
    """,
    expected_output="Pre-deploy validation report with status (PASS/FAIL) and list of config mismatches",
    agent=devops_agent,
)

This can block a deploy pipeline on FAIL, or surface warnings for review. Because it is running local inference, it adds minimal latency to the pipeline — no round-trip to an external API.

See the demo for a live walkthrough of this workflow in Clanker Cloud.

---

Security and Compliance

Local Hermes is one way to avoid sending model prompts to OpenAI or another cloud-model provider. It is only one part of the regulated-system data-flow and control review.

Reduced model egress. Hermes inference and the MCP connection run locally. Raw provider credentials stay local, but provider API calls and ordinary Clanker Cloud account, security, download, or update traffic remain. Hosted features must stay disabled if their paths are outside the approved boundary.

Credential isolation. The agent code never handles cloud provider credentials. It calls the Clanker Cloud MCP server, which handles auth internally. This means your AWS access keys, Kubernetes service account tokens, and other credentials are never present in agent prompts or logs.

Auditability. Local inference can make model execution easier to inspect, but teams must configure and verify the required logs, retention, access controls, and tamper protection. The product does not create a complete compliance audit trail automatically.

Regulated-use status. Local inference can reduce a transfer path, but it does not make Standard suitable for PHI, criminal-justice information, classified data, or a government-authorized system. Hosted regulated use requires the applicable signed DPA, BAA, CJIS or government terms and a separately verified active protected environment. A local-only customer configuration still requires its own legal and security assessment.

---

FAQ

What is Hermes 3 and how does it compare to GPT-4?

Hermes 3 is an open-source instruction-following and function-calling model by NousResearch, built on Meta's Llama 3.1 base. Running it through local Ollama avoids a cloud-model API key and per-token model charge and can keep model prompts local. That does not eliminate cloud-provider calls, account traffic, or the need to assess every enabled feature for regulated use.

Can I run an AI DevOps agent locally without sending data to the cloud?

Hermes inference and Clanker Cloud's MCP connection can run locally, and raw provider credentials can stay in the desktop credential chain. Cloud-provider calls still leave the machine for those providers; account and update traffic may occur; and hosted inference, sandboxes, voice, or web remote control create additional paths. Verify the configured network flow rather than assuming zero egress.

How do I use Hermes with Clanker Cloud?

Install Ollama, pull Hermes 3 with ollama pull nous-hermes-3, and install the Clanker Cloud desktop app. Connect your cloud providers in the desktop app, note the MCP server endpoint, and configure your agent framework (LangChain or CrewAI) to use Hermes as the LLM and Clanker Cloud's MCP endpoint as the tool source. Full setup is covered in the walkthrough above, and detailed documentation is at docs.clankercloud.ai.

What agent frameworks work with Hermes 3?

Any framework that accepts an OpenAI-compatible API endpoint works with Hermes via Ollama. This includes CrewAI, LangChain, AutoGen, and LlamaIndex. The most common pattern is configuring Ollama(model="nous-hermes-3", base_url="http://localhost:11434") as the LLM within the framework. CrewAI's LLM class also supports LLM(model="ollama/nous-hermes-3") directly. For MCP tool integration, LangChain's MCP adapters or the CrewAI MCP toolset handle the protocol bridge to Clanker Cloud. See the for-agents page for integration patterns and the FAQ for common setup questions.

---

Get Started

The Clanker Cloud desktop app is free during beta. Connect your first cloud provider, start the MCP server, and wire in Hermes via the snippets above.

Full documentation — including MCP server configuration, supported providers, and BYOK model setup — is at docs.clankercloud.ai.

For teams already running infrastructure workflows, the AI DevOps for teams guide covers multi-agent patterns, role-based access, and shared workspace setup.