Skip to main content

The Missing Layer in Every Agent Harness

· 16 min read
Ben Bartholomew
Ben Bartholomew
Hindsight Team

The Missing Layer in Every Agent Harness

Pick any modern agent harness — Claude Code, Cursor, OpenClaw, Hermes, Codex, Aider, Cline, Roo Code, Pipecat. They have all gotten dramatically better in the last 18 months. They have file access. They have shell access. They have MCP servers. They have browser control. They have rich IDE integrations and slash commands and skills and subagents.

What almost none of them ship with is memory that learns.

The harness is the part of an agent system that wraps the model: it manages the loop, exposes tools, coordinates subagents, handles permissions, and renders output. It is the infrastructure between you and the LLM. And the harness layer is where long-term memory most obviously belongs — but it is also the layer where most harnesses have stopped at the filesystem and a static context file and called it done.

TL;DR​

  • Agent harnesses (Claude Code, OpenClaw, Hermes, Cursor, Codex, etc.) have invested heavily in tools, MCP, and IDE integrations
  • They have largely skipped the memory layer, leaving every session to start from zero
  • Workarounds like CLAUDE.md, AGENTS.md, and rules files help, but they are static and require manual upkeep
  • A real memory layer in the harness means the agent learns from prior sessions automatically — preferences, decisions, conventions, dead ends
  • Harnesses like Hermes (native memory plugin) and OpenClaw (memory plugin with autoRecall) are leading the shift; others are catching up
  • Hindsight provides the memory layer that any harness can integrate through a simple API

What A Harness Actually Is​

It helps to be precise about the layers, because "agent" gets used to mean a lot of different things.

The model is the LLM. Claude, GPT, Gemini, Qwen, whatever you point the harness at. It is stateless. It takes tokens in and emits tokens out.

The harness is the runtime around the model. It owns the agent loop, the tool registry, the permissions, the prompt assembly, the rendering. Claude Code is a harness. OpenClaw is a harness. Cursor is a harness with a chat UI bolted onto an editor. Hermes is a harness oriented around long-running agents.

The agent is what emerges when a harness drives a model against a goal using tools. It is not a thing you install — it is the runtime behavior of the harness plus the model plus the prompt. The framing Agent = Model + Harness has become the standard way to describe this split — see OpenAI on harness engineering, Martin Fowler on harness engineering for coding agents, and LangChain's anatomy of an agent harness for three anchoring takes.

When people say "agents are forgetful," what they usually mean is "harnesses do not learn." The model was never going to remember anything across sessions; that was always going to be the harness's job. The harness does have a memory primitive — the filesystem, plus a static instructions file that gets re-read every session — but neither of those compounds. Nothing the agent learns from the work feeds back in.

What Harnesses Have Already Solved​

Before talking about the gap, it is worth acknowledging how much progress harnesses have made on everything else.

Tools. Every modern harness has rich tool calling. File reads, edits, writes, shell commands, web fetch, web search. The tool registry is no longer the bottleneck.

MCP. Model Context Protocol gives harnesses a standard way to plug in external systems. Slack, Linear, Gmail, Postgres, custom internal services — any of these can become a tool with a few lines of config. This was a major leap in 2025.

IDE integration. Claude Code, Cursor, and Codex live inside or alongside the editor. They see what you are looking at, they edit in place, they share buffers, they understand selection.

Browser and OS control. OpenClaw and similar harnesses can drive a browser, click through a UI, fill forms, take screenshots. The agent's effective surface area now extends well beyond the file system.

Subagents and skills. Claude Code has subagents, skills, and slash commands. OpenClaw has plugin architecture. Hermes has native multi-agent orchestration. The harness layer increasingly knows how to delegate, parallelize, and compose.

Permissions and safety. Sandboxes, allow lists, dangerous-action confirmations, hooks. The harness layer is also where guardrails live, and that has matured fast.

That is a lot of progress. The harness used to be a thin loop around the model; now it is a serious runtime.

What it still cannot do well, in almost every case, is learn. The harness can persist files. It can re-read a CLAUDE.md every morning. What it cannot do is accumulate understanding from the work itself and have that understanding shape the next session.

The Memory Gap​

Open a new session in any of the harnesses above and you will hit the same wall.

The harness knows what tools you have. It does not know what you decided last week about which database to use.

The harness can read your codebase. It does not know that two sessions ago, the team rejected a refactor pattern and explained why.

The harness can call your Linear MCP server. It does not know which tickets you have already triaged this morning, or which bugs you tend to push to the next sprint.

The harness can take a screenshot. It does not know that you prefer terse explanations and hate it when an agent restates the question back to you.

That information existed. It was generated through interaction. The harness never captured it, so the next session arrives blank.

You can feel this most acutely in coding harnesses, because coding is iterative and contextual. But it shows up everywhere: support harnesses that re-onboard the same customer every conversation, voice harnesses that lose all context the moment the call ends, multi-agent harnesses where one instance discovers something useful and the others never find out.

The Workarounds — And Why They Are Not Enough​

The harness ecosystem has settled on a few partial answers. They help. None of them is learning memory.

The filesystem​

Most harnesses treat the filesystem as the de facto memory layer — write a file, read it back next session. It is the most foundational primitive a harness has, and for a lot of tasks it works fine. The limit is that the harness is doing the writing and the reading, but no one is doing the organizing. There is no synthesis, no relevance ranking, no decay. Files pile up. The agent has to know exactly which file to open. It is a hard drive, not a memory.

Static context files (CLAUDE.md, AGENTS.md, .cursorrules)​

Most coding harnesses now support a project-level instruction file. Claude Code reads CLAUDE.md. Codex reads AGENTS.md. Cursor reads .cursorrules. The file is appended to the system prompt every session.

These are useful. They are also static. Someone has to write them, keep them current, prune them when they grow stale, and remember to update them when something changes. They capture intent, not interaction. Nothing the agent learns from the work feeds back into the file.

If you ever look at a CLAUDE.md that has not been touched in a few weeks, you see the failure mode. Half of it is wrong. Half of what should be there is missing. The harness keeps reading it dutifully.

Session compaction and summaries​

Long sessions get auto-compacted. Some harnesses summarize at session end. This is a context-window management trick, not memory — the summary lives inside the session that produced it, and the next session starts without it.

Manual recall​

The most common pattern is the user pasting context back in. "Here's what we decided last time." "Here's the file we ended on." "Here's the error from the last run." That is memory implemented in the user's brain and clipboard.

Document retrieval​

Pointing the harness at a doc store (RAG over /docs, codebase indexes, ticket archives) is genuinely valuable, but it solves the reference-material problem. It does not capture lived interaction history — what happened in this conversation, with this user, on this project, that did not get written down anywhere.

What Changes When The Harness Has Memory​

Once memory is a first-class layer in the harness, the day-to-day shifts in ways that are easy to underestimate.

The agent stops re-asking which package manager you use. It stops suggesting the abstraction you explicitly rejected last week. It stops re-explaining the deployment process to you. It starts the next session with context from the last one already loaded — not pasted in, not summarized, but actually retained and recalled.

Multi-agent setups change more dramatically. If a Slack agent and a Discord agent and an in-IDE agent share the same memory bank, what one of them learns the others know. A user who asked the Slack bot a question on Monday gets continuity from the IDE agent on Tuesday.

Long-running agents — voice, support, automation — stop losing the relationship every time the session ends.

And critically, the harness becomes capable of compounding improvement. Each session leaves the system slightly more useful than it found it. That is the property stateless harnesses cannot have at all.

How Specific Harnesses Are Approaching This​

The shift is uneven. Some harnesses have started building memory directly. Others are happy to integrate an external memory layer through MCP or plugins.

Hermes​

Hermes ships a native memory plugin. The pattern is the cleanest example of memory-as-harness-feature — the plugin handles retention and recall as part of the agent loop, not as an external concern. A hermes memory setup wizard in a recent release makes the integration nearly one-step.

Hermes also takes the idea a step further than most harnesses: it can create its own skills. When the agent encounters a workflow it expects to repeat, it can write a new skill — a reusable, callable capability — and add it to its own toolbox for future sessions. That is a real harness-level differentiator, and it is closer to memory than it first appears: a skill is procedural memory, the kind that captures how to do something rather than what happened. Pair it with declarative memory (facts, decisions, preferences from a memory layer like Hindsight) and you have both halves of how a working agent should remember. It is also one of the clearest precursors to self-driving agents — a harness that can extend its own capabilities is a harness that can iterate without you.

OpenClaw​

OpenClaw treats memory as a first-class plugin too. A typical config looks like this:

plugins:
  hindsight:
    bank: openclaw
    autoRetain: true
    autoRecall: true

Retention happens during the work and recall happens before the next response — no prompting required. Because OpenClaw acts as a gateway — it speaks Slack, Discord, Telegram, browser, IDE, all at once — the memory layer is what makes the gateway coherent across surfaces. Without it, each surface forgets independently.

Claude Code​

Claude Code does not have native long-term memory yet, but it integrates memory layers through hooks, MCP servers, and skills. The hindsight-memory plugin uses the hook architecture directly: SessionStart for health checks, UserPromptSubmit for auto-recall, Stop for auto-retain. CLAUDE.md covers the static slice; the MCP-backed memory layer handles the dynamic slice — the things that should accumulate without anyone editing a file.

Codex​

Codex has AGENTS.md for static context and is increasingly used in multi-agent configurations where shared memory between Codex instances matters more than memory inside any single one. The memory layer becomes the substrate that lets multiple Codex instances behave like a team rather than parallel strangers.

Cursor, Cline, Roo Code, Aider​

These IDE-resident harnesses generally rely on rules files for static context and have minimal native memory. The integration path is usually MCP. Roo Code has a memory cookbook example; the pattern works in any of them.

Pipecat (voice)​

Voice harnesses are where statelessness hurts the most. The call ends, and the agent forgets the user. Pipecat with a memory layer can carry context across calls — the user does not have to re-explain anything when they call back next week.

Superagent, Haystack, LangGraph, CrewAI, Pydantic AI​

The orchestration frameworks all support pluggable memory; most have integration paths to external memory layers. The pattern is consistent: the framework owns the agent loop, the memory layer owns retention and recall.

The trend across all of these is the same. Either the harness builds memory in (Hermes), or the harness exposes a clean integration point (everyone else) and a memory layer plugs in.

What A Memory Layer Has To Do For A Harness​

A useful harness-level memory system is not just a key-value store. It has to do four things well, and they map directly onto how harnesses run.

Retain selectively from the agent loop​

The harness sees every tool call, every model response, every user message. Most of that is noise. Memory has to capture the durable parts — decisions, preferences, facts, patterns — and skip the rest. Retention happens during the loop, not after it.

Recall before the model is called​

The harness already assembles a prompt before each model call. That is the natural place to inject relevant memory — not all memory, just what is relevant to the current task. Recall has to be cheap enough to run on every turn and precise enough to not flood the context.

Reflect across accumulated experience​

Some questions are not answered by recalling one fact. They are answered by synthesizing across many. A harness with memory should be able to ask "what do we know about this codebase" and get a synthesized answer, not a list of fragments.

Scope correctly across users, projects, and surfaces​

A coding harness needs per-project memory. A multi-tenant support harness needs per-user memory. A swarm of OpenClaw agents needs shared memory. A team using Codex needs shared-by-default with optional isolation. The memory layer has to support all of these patterns without forcing the harness to invent its own scoping logic.

These are exactly the design points Hindsight optimizes for. Retention, recall, reflection, and scoping are the four primitives, and they are exposed through an API and integrations that any harness can consume.

Why Memory Matters For Harnesses Now​

A year ago you could argue that memory was a nice-to-have because harnesses were still figuring out tools. That argument does not hold anymore.

Harnesses have tools. They have IDE integration. They have MCP. They have subagents and skills and permissions. The remaining gap between "an agent that is impressive in a single session" and "an agent that gets better at your work over months" is almost entirely memory.

The harnesses that figure this out — either by building it in, like Hermes, or by integrating cleanly, like OpenClaw and Claude Code — will feel meaningfully different to use within a few weeks of accumulated interaction. The ones that do not will keep being impressive in demos and frustrating in week three.

How Hindsight Fits​

Hindsight is designed to be the memory layer any harness can drop in. It is API-first, MCP-compatible, and ships with integrations for the major frameworks and harnesses.

In practice, integrating it into a harness looks like this:

  • The harness retains relevant context to a memory bank as the loop runs (autoRetain, or explicit retain calls)
  • Before each model call, the harness recalls relevant memories and injects them into the prompt (autoRecall, or explicit recall calls)
  • The memory bank scopes correctly — per project, per user, per team, or shared across a swarm — using memory bank configuration
  • Periodically, accumulated memories synthesize into observations and mental models that the harness can query directly when it needs synthesis, not just recall

The harness keeps doing what it is good at: running the loop, calling tools, managing permissions, rendering output. The memory layer handles the part the harness was never designed to handle.

You can run Hindsight with Hindsight Cloud for the fastest path, or self-host if data needs to stay in your environment. Either way, the integration into a harness is the same.

Recap​

  • Modern agent harnesses have invested heavily in tools, MCP, IDE integration, and orchestration
  • The memory layer is the most obvious remaining gap, and it is the layer that determines whether the agent improves with use
  • Static workarounds (CLAUDE.md, AGENTS.md, rules files) help but cannot capture lived interaction
  • Hermes ships native memory; OpenClaw, Claude Code, Codex, and the orchestration frameworks integrate memory as a layer
  • A real harness-level memory system has to retain selectively, recall precisely, reflect across experience, and scope correctly

If your harness has every tool you need and you still feel like the agent is starting over every morning, the gap is not the model and it is not the tools. It is the missing memory layer.

Further Reading​

Next Steps​