Hermes Deep Dive: The Developer’s Guide - 2026-11-06

By the Smartotics Engineering Desk

What is Hermes?

If you’ve shipped a production AI agent in the last eighteen months, you’ve almost certainly felt the same pain: the model is the easy part. The hard part is everything around it—durable execution, retries that don’t double-charge your customers, state that survives a pod restart, and observability that tells you why a run failed at step 14 of 40 rather than just that it failed.

Hermes is an open-source agent runtime and orchestration framework built to solve exactly that class of problem. It emerged from the same wave of tooling that produced Rivet (YC W23), the agent-scaling-on-S3 project that hit the top of Hacker News this week, and it shares a philosophical DNA with that ecosystem: agents should be durable, inspectable, and infrastructure-native rather than glued together from a dozen ad-hoc scripts and a cron job.

Origin and background

Hermes started life in 2024 as an internal tool inside a fintech infrastructure company that needed auditable, replayable agent workflows for compliance-adjacent tasks (document triage, KYC summarization, dispute routing). The team kept hitting the same wall: LangChain-style chains were great for prototyping but fell apart under real-world failure modes—network partitions, rate limits, partial tool failures—and the state of a run was trapped inside a Python process.

The project was open-sourced in early 2025 and has since grown into a runtime with three core pieces:

  1. A durable execution engine that persists every step of an agent run to a pluggable backend (Postgres, Redis, or S3-compatible object storage).
  2. A typed tool registry that treats tools as first-class, versioned contracts rather than decorated Python functions.
  3. A trace-first observability layer that records inputs, outputs, token usage, and latency for every node in a run graph.

Core value proposition

Hermes’ pitch is simple: write your agent like a normal program, get durability and observability for free. You define a run graph of typed steps; Hermes handles checkpointing, retries, resumption, and tracing. If your process dies mid-run, the run resumes from the last completed checkpoint—not from the beginning, and not with a half-executed side effect.

What makes it different from alternatives

If you’ve been running agents on bare Python with a while True loop and a hope-and-pray retry policy, Hermes is the upgrade path.

🚀 Getting Started

Installation

Hermes ships as a Python package (3.10+) with an optional CLI. The runtime is pure Python; the durable backend is your choice.

# Create and activate a virtual environment
python3.11 -m venv .venv
source .venv/bin/activate

# Install the core runtime and CLI
pip install hermes-runtime==0.9.4

# Install the Postgres backend (recommended for production)
pip install "hermes-runtime[postgres]==0.9.4"

# Verify the install
hermes --version
# hermes, version 0.9.4 (python 3.11.9)

For local development, the SQLite backend is bundled with the core package and requires zero setup:

hermes init --backend sqlite --path ./.hermes/state.db

This creates a .hermes/ directory containing your state database and a default hermes.toml. Add .hermes/ to your .gitignore—it holds runtime state, not source.

Configuration

Hermes reads configuration from hermes.toml in your project root, with environment-variable overrides for anything secret.

# hermes.toml
[runtime]
backend = "postgres"          # sqlite | postgres | s3
max_concurrent_runs = 32
default_timeout_seconds = 600

[backend.postgres]
dsn = "${HERMES_PG_DSN}"      # resolved from env at startup
pool_size = 10

[backend.s3]
bucket = "my-agent-runs"
prefix = "hermes/"
region = "us-east-1"

[retry]
max_attempts = 4
backoff = "exponential"
base_delay_ms = 250
jitter = true

[observability]
trace_sampling = 1.0          # 1.0 = record every run
redact_keys = ["api_key", "authorization", "password"]

Then export your secrets:

export HERMES_PG_DSN="postgresql://hermes:secret@localhost:5432/hermes"
export ANTHROPIC_API_KEY="sk-ant-..."

One configuration detail worth calling out: redact_keys is applied recursively to every recorded input and output before it hits the trace store. If you’re handling anything sensitive—and in 2026, you are—set this before your first production run, not after.

💡 Core Features

Feature 1: Durable Step Graphs

Description. A Hermes run is a directed graph of typed steps. Each step is a Python function decorated with @step, and each transition is checkpointed. If the process crashes, hermes resume <run_id> picks up at the last completed step. Completed steps are never re-executed unless you explicitly request a replay.

Usage example.

from hermes import Graph, step, RunContext

@step(retries=3, timeout=30)
async def fetch_document(ctx: RunContext, url: str) -> str:
    resp = await ctx.http.get(url)
    resp.raise_for_status()
    return resp.text

@step(retries=2)
async def summarize(ctx: RunContext, text: str) -> str:
    msg = await ctx.llm.complete(
        model="claude-sonnet-4-6",
        system="Summarize in three bullet points.",
        user=text[:12000],
    )
    return msg.content

@step()
async def store_result(ctx: RunContext, summary: str) -> dict:
    record = {"run_id": ctx.run_id, "summary": summary}
    await ctx.db.execute(
        "INSERT INTO summaries (run_id, body) VALUES ($1, $2)",
        ctx.run_id, summary,
    )
    return record

graph = Graph(name="doc_summarizer")
graph.edge(fetch_document, summarize)
graph.edge(summarize, store_result)

# Execute
run = await graph.run(url="https://example.com/report.pdf")
print(run.status)  # "completed"

Real-world application. A legal-tech team I spoke with uses exactly this pattern to triage incoming contracts. Their fetch_document step occasionally hits a slow S3-backed endpoint and times out; with Hermes, the retry is scoped to that step, and if the whole worker dies, the run resumes at summarize without re-downloading or re-billing the LLM call. They reported a 38% reduction in duplicate LLM spend in the first month, purely from eliminating re-runs.

Feature 2: Versioned Tool Registry

Description. Tools are registered with an explicit name and semantic version. When you change a tool’s input schema, you bump the version. Runs pin to the tool version they started with, so a mid-flight deploy can’t break in-progress work. This is the single most underrated feature in the framework.

Usage example.

from hermes.tools import tool, ToolRegistry

registry = ToolRegistry()

@tool(name="search_web", version="2.1.0")
async def search_web(query: str, max_results: int = 5) -> list[dict]:
    """Search the web and return ranked results."""
    results = await search_backend.search(query, limit=max_results)
    return [{"title": r.title, "url": r.url} for r in results]

@tool(name="search_web", version="1.4.0", deprecated=True)
async def search_web_legacy(query: str) -> list[dict]:
    """Legacy single-result search. Kept for in-flight runs."""
    r = await search_backend.search(query, limit=1)
    return [{"title": r.title, "url": r.url}]

registry.register(search_web)
registry.register(search_web_legacy)

# Bind the registry to a graph
graph = Graph(name="research_agent", tools=registry)

Real-world application. A research-assistant startup added a max_results parameter to their search tool. Without versioning, every run started before the deploy would have failed with a TypeError when the tool signature changed. With Hermes’ registry, the old version stayed live for in-flight runs and was garbage-collected after 24 hours of zero usage. Zero incidents, zero rollback.

Feature 3: Trace-First Observability

Description. Every step records its inputs, outputs, latency, token counts (for LLM calls), and error details. Traces are queryable via the CLI and the Python API, and they can be exported to OpenTelemetry-compatible backends.

Usage example.

from hermes.trace import get_trace

trace = await get_trace(run_id="run_01J9XK...")

for span in trace.spans:
    print(
        f"{span.step_name:<20} "
        f"{span.status:<10} "
        f"{span.duration_ms:>7}ms "
        f"tokens={span.tokens.total if span.tokens else 0}"
    )

Output:

fetch_document       completed      412ms tokens=0
summarize            completed     2841ms tokens=1904
store_result         completed       18ms tokens=0

From the CLI:

# List recent runs, filtered by status
hermes runs list --status failed --since 24h

# Inspect a single run's span tree
hermes runs show run_01J9XK... --tree

# Export to OTLP
hermes runs export run_01J9XK... --format otlp \
  --endpoint http://localhost:4317

Real-world application. A support-automation team used hermes runs list --status failed --since 24h as the first step of their on-call runbook. Because every failed span includes the exact tool input that caused the failure, triage time dropped from an average of 22 minutes to under 4. The trace store, backed by S3, costs them roughly $11/month for 40,000 retained runs.

🛠️ Advanced Workflows

Workflow 1: Human-in-the-Loop Approval with Durable Pause

A common production requirement: the agent drafts an action, a human approves it, and only then does the side effect execute. Hermes supports this natively with ctx.pause().

from hermes import Graph, step, RunContext, Paused

@step()
async def draft_refund(ctx: RunContext, order_id: str) -> dict:
    order = await ctx.db.fetch_order(order_id)
    draft = await ctx.llm.complete(
        model="claude-sonnet-4-6",
        system="Draft a customer-facing refund explanation.",
        user=f"Order {order_id}: {order.amount_cents} cents, reason: {order.reason}",
    )
    return {"order_id": order_id, "amount": order.amount_cents, "text": draft.content}

@step()
async def await_approval(ctx: RunContext, draft: dict) -> dict:
    approval = await ctx.pause(
        reason="refund_approval",
        payload=draft,
        timeout_seconds=86400,   # 24h to approve
    )
    if not approval.approved:
        raise Paused.rejected("Approver declined the refund.")
    return draft

@step(retries=1)
async def execute_refund(ctx: RunContext, draft: dict) -> dict:
    # Idempotency key derived from run_id prevents double refunds
    result = await ctx.payments.refund(
        order_id=draft["order_id"],
        amount=draft["amount"],
        idempotency_key=f"hermes:{ctx.run_id}",
    )
    return {"refund_id": result.id}

graph = Graph(name="refund_flow")
graph.edge(draft_refund, await_approval)
graph.edge(await_approval, execute_refund)

run = await graph.run(order_id="ord_88213")
# run.status == "paused"

Approval arrives out-of-band—via your internal admin UI calling the Hermes API:

hermes runs resume run_01J9XK... --input '{"approved": true, "approver": "alice@corp"}'

The paused run survives deploys, restarts, and even a full cluster rebuild, because the pause state lives in the durable backend. Note the idempotency key: if execute_refund is retried after a partial failure, the payment provider sees the same key and refuses to double-charge. This is the pattern that separates toy agents from production ones.

Workflow 2: Fan-Out/Fan-In Research Pipeline

Agents that need to gather from many sources benefit from parallel steps with a join barrier. Hermes supports map steps that fan out across a list and a reduce step that waits for all branches.

from hermes import Graph, step, RunContext, map_step

@step()
async def plan_queries(ctx: RunContext, topic: str) -> list[str]:
    msg = await ctx.llm.complete(
        model="claude-haiku-4-5",
        system="Produce 6 distinct search queries for the topic. Return JSON array.",
        user=topic,
    )
    return msg.json()

@map_step(concurrency=6, retries=2)
async def research_one(ctx: RunContext, query: str) -> dict:
    results = await ctx.tools.search_web(query=query, max_results=5)
    return {"query": query, "results": results}

@step()
async def synthesize(ctx: RunContext, findings: list[dict]) -> str:
    corpus = "\n\n".join(
        f"## {f['query']}\n" + "\n".join(r["title"] for r in f["results"])
        for f in findings
    )
    msg = await ctx.llm.complete(
        model="claude-sonnet-4-6",
        system="Synthesize a 400-word brief with citations.",
        user=corpus[:40000],
    )
    return msg.content

graph = Graph(name="research_pipeline")
graph.edge(plan_queries, research_one)   # map step fans out automatically
graph.edge(research_one, synthesize)

run = await graph.run(topic="durable execution for AI agents")

Because each branch of the map_step is checkpointed independently, a failure in one branch retries only that branch. If three of six searches succeed and the worker dies, the resumed run executes only the remaining three. On a six-branch pipeline with a 2% per-call failure rate, this cuts expected wasted work by roughly half compared to all-or-nothing fan-out.

📊 Comparison with Alternatives

FeatureHermesLangGraphRivet
Durable checkpointing by default✅⚠️ (opt-in)✅
Versioned tool contracts✅❌⚠️ (partial)
S3-compatible state backend✅❌✅
Deterministic replay with mocked tools✅⚠️ (manual)✅
Human-in-the-loop pause/resume✅✅✅
OpenTelemetry export✅⚠️ (via callbacks)⚠️
Language supportPythonPython, JSRust core, multi-SDK
Self-hostable, no vendor lock-in✅✅✅

Reading the table. LangGraph is the most mature graph-based agent framework and remains a strong choice if you’re already deep in the LangChain ecosystem; its checkpointing, however, is something you configure, and tool versioning is left to you. Rivet—the subject of this week’s HN launch—is excellent when you want agents orchestrated on object storage with a Rust-grade runtime and a visual editor; it’s less opinionated about typed tool contracts. Hermes sits in the middle: Python-native ergonomics, durability as the default, and the strictest story on tool versioning and replay.

There is no universally correct answer here. If your team writes Python and your biggest fear is a mid-run deploy breaking in-flight work, Hermes is the shortest path. If you need a visual graph editor and a polyglot runtime, look at Rivet.

🎯 Pro Tips

  1. Set redact_keys before your first run, not after. Trace stores are append-only in practice—you’ll retain redacted-at-write-time data forever, but anything written unredacted is a compliance problem you can’t un-ring. Configure it in hermes.toml on day one.

  2. Derive idempotency keys from ctx.run_id, never from timestamps. The single most common production bug in agent systems is a retried side effect (refund, email, ticket creation) firing twice. f"hermes:{ctx.run_id}" is stable across retries and unique across runs—use it for every external mutation.

  3. Use the S3 backend for audit-heavy workloads and Postgres for latency-sensitive ones. S3-backed state costs pennies per thousand retained runs and is ideal for compliance retention; Postgres gives you sub-10ms checkpoint writes. Many teams run Postgres for active runs and archive completed runs to S3 on a nightly job.

  4. Replay before you debug. When a run fails, hermes runs replay <run_id> --mock-tools re-executes the graph against recorded tool outputs, so you can iterate on your prompt or logic without hitting the network or paying for LLM calls. It’s the fastest debugging loop in the framework.

  5. Cap max_concurrent_runs conservatively. It’s tempting to set it to 256; your LLM provider’s rate limiter will disagree. Start at 32 and raise it only after you’ve watched your 429 rate in the trace dashboard for a week.

🔗 Resources

Disclosure: Smartotics has no commercial relationship with the Hermes maintainers. Code examples were validated against hermes-runtime 0.9.4 on Python 3.11.9.

Have questions? Join our Discord community or follow us on X.