Skip to content
All tags

#harness-engineering

36 posts

Learning Agent Design from Mature Coding Agents: Series Overview — Reading Five Codebases to Build My Own

I'm building my own Python coding agent called looplane. This series dissects the source code of five mature projects — pi, oh-my-pi, opencode, codex, and claude-code — topic by topic, while also comparing them with Looplane's current TUI, external CLI runtimes, local gateway, usage/OTel/session tooling, and Cloudflare slice. Every post follows a fixed five-part structure: design problem → how five projects do it → looplane's choice → academic grounding → improvement roadmap, with evidence cited at file#symbol level.

Learning Design from Mature Coding Agents (6): The ModelProvider Abstraction — Why Wrapping an SDK Is Not Enough

Wrapping an SDK directly buys you three walls within months: usage fields that don't agree, error semantics tied to SDK exception types, and tool-call formats that change per provider. All five reference projects separate 'wire protocol' from 'provider identity' as independent dimensions. Looplane goes further with pydantic canonical contracts (Message/ToolCall/Usage/ModelTurn) plus six protocol adapters, forces the OpenAI SDK's built-in retries to zero, and routes every failure through a classified ProviderErrorKind before any retry policy sees it. Its provider table is deliberately copied from pi's packages/ai — lineage, not coincidence.

Learning from Mature Coding Agents (29): OS-Level Sandboxing

An OS sandbox is the kernel boundary beyond path policy. looplane now ships a fail-closed CommandSandbox: sandbox-exec on macOS, Landlock plus seccomp on Linux, and exit 126 when containment cannot be proven. Coverage still focuses on verification commands, and external CI confirmation remains open.

Learning Design from Mature Coding Agents (7): Provider Retry Policy — From One 5xx to Bounded Retry and Fallback

Intermittent NVIDIA NIM 500s exposed Looplane's early gap: classified errors with no retry consumer. SDK retries are now disabled; the harness gives each candidate up to five attempts with jittered exponential backoff and capped Retry-After handling, then can move to an explicitly configured fallback model. Both model.retry and model.fallback enter the event log.

Learning from Mature Coding Agents (3): Workspace Isolation and Path Policy

Looplane's disposable clone and SafePathPolicy protect the source repo. `--sandbox-checks` can now wrap verification commands with macOS sandbox-exec, Linux bubblewrap, or Landlock, while Cloudflare provides a separate bounded Sandbox slice. Network policy, external-runtime coverage, and production hardening are not yet consistent across those backends.

Same Name, Different Layer: meta-harness, ACP, HarnessAgent and Flue

meta-harness means two things: Databricks' control plane and Stanford's outer-loop optimizer. This post uses a four-layer model (MCP/ACP/Runtime/meta-harness) to place Omnigent, Zed ACP, Vercel HarnessAgent and Cloudflare Flue.

Learning Design from Mature Coding Agents (10): Edit Tool Trade-offs — unified diff, exact edit, hashline, and whole-file

LLMs break unified diffs on bookkeeping: wrong hunk counts, hallucinated context lines. The five reference projects split into two camps — simplify the diff grammar (Codex drops line numbers), or drop diffs entirely (Claude Code/Pi/OpenCode exact replace); OMP goes further by binding read state into the format via hash anchors. looplane took the minimal-intervention path: keep the guarded apply_patch, add a zero-fuzzy replace_text, and its qwen3:4b eval went from stable failure to 5/5.

Learning Design from Mature Coding Agents (22): The Gateway Pattern — Turning Any Provider into an OpenAI-Compatible Endpoint

The ecosystem treats /v1/chat/completions as the lingua franca, but your providers don't all speak it. The five reference projects split into three camps: pi and OpenCode make the client speak every dialect natively so no gateway is needed; OMP builds a real protocol translator (foreign wire → neutral context → provider adapter, no raw passthrough); Codex and Claude Code run proxies that translate nothing and exist purely to force traffic through a controllable path. Looplane copies OMP's boundary but narrows it to one wire in, one out: strictly parse OpenAI Chat into a canonical contract, then dispatch to any ModelProvider — and along the way hit a cross-event-loop client-close bug whose lesson is that provider lifecycles belong to the ASGI lifespan, not the signal handler.

Learning from Mature Coding Agents (16): Runtime Abstraction and Capability Handshake

Five external CLIs expose five different machine interfaces: JSONL event streams, JSON-RPC handshake, HTTP API, ACP, stream-json. The right way to support them is not one interface that pretends they're identical — it's a narrow runtime boundary plus an honest capability matrix. Availability means installed, not authenticated; protocol drift fails closed.

Learning from Mature Coding Agents (11): Sandboxes and Remote Execution — Deploying on Cloudflare Sandbox

A local sandbox limits the blast radius of an agent on your machine; a cloud sandbox is about moving code safely onto someone else's machine. All five mature projects solve the first problem; only looplane actually deployed the second. Lessons from production: mocks can't catch SSE framing, green CI can't catch a stale wheel, and cleanup paths deserve timeouts just as much as success paths.

Learning from Mature Coding Agents (12): Can Small Models Code? — Capability Boundaries and Eval Discipline

Small models don't fail at reasoning first — they fail at format stability: tool-call JSON, diff hunk arithmetic, and context budgets all break. The mature harnesses build evals on real model behavior (pi's model-backed evals, OMP calibrating benchmarks from real session logs, Codex even relaxing its parser for weaker models). looplane picks the narrowest but hardest path: one fixture, five real Ollama runs, a manifest declaring exactly which files and patch fragments count as success — and M2's failure kept verbatim as evidence. Never pass mock off as E2E; never spin partial success into full passes.

Learning Design from Mature Coding Agents (8): The Right Way and the Wrong Way to Use Subscriptions — OAuth and Credential Boundaries

The five reference projects split into three camps on subscription auth. Codex and Claude Code implement OAuth only for their own official clients and store tokens in the OS keyring. pi and OMP directly reuse Claude Code's client ID to implement Pro/Max OAuth — technically feasible, but Anthropic's docs explicitly bar third parties from offering claude.ai login without approval. OpenCode removed its bundled Pro/Max plugins entirely, the cleanest policy precedent in the ecosystem. Looplane's rules: own your grant, never scrape another CLI's credentials, accept third-party OAuth only when the provider clearly supports it, and never copy or forward credentials.

Learning Agent Design from Mature Coding Agents (5): The Verification Gate — Changed Files Isn't Success, Verified Is

None of the five reference projects enforces 'all declared verification commands pass' at the harness level: pi leaves verification to the model, OpenCode and Codex put it in the system prompt, Claude Code uses a separate adversarial verifier subagent but as a soft contract, and only OMP's cleanse actually runs checks from harness code. looplane takes the hardest path: if files changed, every declared verification command must pass before terminal_reason=verified; with no changes, checks don't rerun (no_changes). Whether to verify is decided by code, not by the model.

Antigravity CLI: Google Replaces a 100K-Star Open-Source Tool with a Closed-Source Go Binary

At Google I/O 2026, Antigravity CLI (agy) replaced Apache 2.0 Gemini CLI with a closed-source Go binary. Technical upgrades — multi-agent orchestration, native sandbox, millisecond startup — but free tier cut 98%, open-to-closed source, 28-day transition window. Community reaction was sharp.

Grok Build: xAI's Rust Coding Agent That Uploaded Your Repo Before Going Open Source

Grok Build is xAI's Rust coding agent — 845K LOC, 8 parallel sub-agents, Arena Mode. May 2026 beta, July open-sourced (Apache 2.0) — but the direct trigger for open-sourcing was a privacy incident: it silently uploaded entire repos (including SSH keys, .env files) to Google Cloud Storage at a 27,800x traffic ratio. The exfiltration code remains in the binary, disabled only by a server-side flag.

Muse Code: Meta's First Coding Agent, Trading Training Rights for a 20x Discount

In August 2026, Meta Superintelligence Labs released Muse Code beta. Closed-source static binary, Muse Spark 1.2 model, parallel persistent sub-agents with worktree isolation. The biggest controversy is pricing: Standard at $1.25/$4.25 per M tokens, or Contributor at $0.10/$0.20 — 20x cheaper, but your code enters Meta's training pipeline.

The H2 2026 Harness War: Eight Frameworks Rewriting, Three Model Makers Entering, 110+ CLIs — How to Make Sense of It

In August 2026, it's not just five frameworks moving. Beyond OMP 2, Pi v2, Opencode 2, dsh, and Claude Code, three model makers — Google (Antigravity CLI), Meta (Muse Code), and xAI (Grok Build) — are building coding agents directly. Add Amp, Cline 2.0, and the Codex CLI Rust rewrite, and eight-plus frameworks are undergoing architecture-level changes simultaneously. Factor in 110+ total CLI tools, and H2 2026 is a divergence period for harness methodology. This article analyzes four architectural approaches, one shared direction, and one emerging trust crisis.

DeepSeek Harness (dsh): A Coding Agent Framework That Takes Everything-is-a-Plugin All the Way

DeepSeek Harness (dsh) is DeepSeek's official open-source coding agent framework, released as a v0.1 developer preview on 2026-08-13, accumulating 184,000+ stars in 9 days. Its core is the Cordis plugin kernel — model adapters, tools, agent loop, and UI are all swappable plugins. Four runtime modes, with the ability to use Claude Code and Codex as sub-agents. Web UI first, no native CLI.

OMP 2 (Oh My Pi 2): From Pi Fork to Full Rust Rewrite as an Independent Coding Harness

OMP 2 is no longer a Pi fork. The entire codebase has been rewritten from scratch in Rust, with ~41 crates covering a custom bash engine, GPU-accelerated GUI, embedded CPython 3.14t, gRPC transport, and Kokoro-82M TTS. Currently in pre-release with no stable version yet.

Opencode 2: The Cost of Swapping Bun for Node, Tauri for Electron, and Rebuilding the Entire API

Opencode 2 is a major rewrite led by Anomaly (Dax Raad). Runtime migrated from Bun to Node.js (memory issues), desktop from Tauri to Electron (WebKit perf and Node integration), v1 API intentionally incompatible. New: multi-tab parallel sessions, persistent backend service, HTTP API + SDK. Currently beta, stable estimated ~September 2026. ~200K stars.

Pi v2: AgentHarness API Goes Stable, Earendil Incorporates — Minimalism Enters Its Next Chapter

Pi v0.84.0 (2026-08-06) promotes the AgentHarness v2 API to stable. Lane-based v4 Session model makes operations durable and interruptible. CBOR replaces JSON, Unix sockets replace HTTP. Earendil Inc. (Armin Ronacher's PBC) behind it has secured initial funding. 95.4K stars, still MIT, still minimal.

ai deep-dive

Seven Answers to a Full Context Window, and No Consensus

Chroma's controlled study shows that even when it fits, a full context degrades performance. Coding agent vendors have landed on seven different responses: compact, hand off, prune, defer loading, isolate, train it into the model, or change the unit of work. Amp removed /compact outright, Atlassian argues summarization should be a last resort, and Cursor's A/B test measured a 46.9% token reduction. The three real disagreements come down to what each team is measuring.

ai deep-dive

The 45 Rules of microsoft/AI-Engineering-Coach: An Opinion About Agentic Engineering, Written as Executable Thresholds

A VS Code extension open-sourced by Microsoft employees that reads your local Claude Code / Codex / OpenCode session logs. The real payload is 45 Markdown rules: prompts under 30 characters, sending the next message within 15 seconds of receiving 20 lines of AI code, instruction files over 4,000 bytes — turning 'context engineering' into numbers you can argue with.

Launch Is Where the Work Starts: Enterprise Agent Cases Read Sideways

Salesforce's number from 20,000 deployments: 90% of the work on an agent happens after launch, the reverse of traditional software. Stripe merges 1,300 PRs a week with no human-written code, and credits the environment rather than the model.

The Model Is a Component, the Harness Is the System

Microsoft, OpenAI, Salesforce, Stripe and three others independently say the same thing: reliability comes from the engineering around the model. And 'give the deterministic parts back to code' has been shipped as a product four separate times — Agent Script, Procedures, runtime, blueprints.

ai deep-dive

Digital Employees: Reliability Comes From the Harness, Not the Model

"Digital employee" isn't a technology — it's a pricing and accountability unit. Anthropic's Project Vend had Claude actually run three shops, and found the most effective intervention wasn't a smarter model but forcing it to follow procedures. Their words: "we rediscovered that bureaucracy matters." Gartner estimates only ~130 of the thousands of vendors claiming to be agentic actually are.

ai deep-dive

Loop Engineering: When AI No Longer Needs You to Write Prompts

Loop Engineering is the practice of designing systems that automatically prompt AI agents, rather than prompting them manually. Boris Cherny runs hundreds of agents, Addy Osmani coined the term, and Blake Crosley identified verification cost as the real bottleneck — this article covers primary sources, the five building blocks, applicability boundaries, and criticisms.

ai deep-dive

How Others Use LLMs to Write: Trade-off Notes from Karpathy's LLM-wiki to Multi-Agent Pipelines

A survey of 11 public LLM writing pipelines, distilled into three dominant patterns: multi-agent (researcher -> writer -> critic), Karpathy LLM-wiki (raw + wiki + LLM writes, humans don't), and quality guardrails (technical verifier + never fabricate + brief gate). The Princeton GEO paper (KDD 2024) quantifies the impact: inline citations +28%, adding statistics +33%, quoting source text +41%, keyword stuffing -9%.

ai guide

Codex App Server: How OpenAI Turned an Agent Harness into a Universal Protocol

OpenAI wrapped the Codex harness as a JSON-RPC over stdio App Server, enabling VS Code, JetBrains, Web, and desktop apps to share a single agent loop. Three core primitives: Item, Turn, and Thread.

OpenAI Wrote 1 Million Lines of Code with Codex: Harness Engineering in Practice

An OpenAI internal team spent 5 months with 3 people and 0 lines of hand-written code, delivering a complete product using Codex. This article distills their core lessons on AGENTS.md design, repo-local knowledge bases, architecture enforcement, and entropy management.

product project

quidproquo Blog Improvement Roadmap: Content, Technical Debt, RAG Design, and Harness Infrastructure

Using my own 30+ RAG/Agent posts to audit the blog itself, I identified a prioritized improvement list spanning content quality, site tech, RAG design fixes, harness infrastructure, and AI agent applications — no phases, just priorities.

ai guide

Agent Skills: A Skill Framework That Makes AI Agents Work Like Senior Engineers

Agent Skills is Addy Osmani's open-source collection of 19 production-grade engineering skills that drive AI agents to follow senior engineering discipline through /spec → /plan → /build → /test → /review → /ship commands, instead of cutting corners.

ai guide

How to Use Claude Code Agent Teams? Design Patterns from 6,400+ Agents on GitHub

There are already 6,400+ .claude/agents/*.md files on GitHub. We dissected 4 representative projects — ChemistryTimes (content production pipeline), claude-sub-agent (document-driven development pipeline), agentic (Temporal.io DAG parallel execution), and vs-copilot-multi-agent (hook-enforced memory persistence) — plus ruflo's enterprise-grade swarm architecture, distilling 6 design patterns and 5 practical trends.

Advanced Harness Engineering Patterns: Tool Registry, Guard System, and Checkpoint-Resume

A Harness is more than just an LLM wrapper. Tool Registry manages dynamic tool loading and selection, Guard System establishes a four-layer defense network, and Checkpoint-Resume enables long-running tasks to survive interruptions. These three patterns form the critical infrastructure of production-grade Agent systems.

From Prompt to Harness: The Three Evolutions of AI Engineering

AI engineering has gone through three phases: Prompt Engineering (write better instructions) → Context Engineering (feed the right information) → Harness Engineering (design the entire working environment). Each evolution doesn't replace the previous one — it operates at a higher level of abstraction.

ai guide

Phil Schmid: Why Agent Harness Is the Most Important Thing in 2026

The model is the CPU, the harness is the operating system, and the agent is the application. No matter how powerful a model is, without a good harness it's just a demo. Phil Schmid argues that harness is the most critical infrastructure in AI engineering for 2026.