CS2680 Lecture 8 (2026-10-07) has two halves. The first takes apart an agent's bill: cost is a product of users × sessions per user × turns per session × steps per turn × tokens per step × price per token, and the last four are the ones you can cut. The students' own Claude Code logs show 97.6% of tokens are cached re-reads, so keeping the prefix stable matters more than keeping the prompt short. Speed is set by the critical path, not the sum of every step. The second half compares three harness architectures (monolithic, modular, microkernel) and the design choices of eight open-source and commercial harnesses. This post works one bill by hand: put a timestamp at the start of the system prompt and the same task goes from 23.6 units back to 65 or more.
To cut the input tokens resent every turn, agent platforms often truncate, clear, or summarize old tool results, and skill instructions, write receipts, and error messages get cut along with them. The Complexity Trap shows observation masking can halve cost, yet Gemini 2.5 Flash with thinking still dropped from 40.4% to 36.4% solve rate. Measure savings against full history and fixes against the current setup, and judge by total cost per successful task, not shorter per-turn input.
Claude's docs say tool-selection accuracy degrades past 30–50 tools, and both Claude and OpenAI now load tools on demand via tool search. But finding a tool isn't finishing the task: on ToolRet's 43k-tool benchmark, no retriever put the complete required toolset in its top 10 more than 35% of the time. This post splits the problem into five failure points, maps seven approaches onto them, and lays out an evaluation.
KV Cache stores the keys and values already computed so decode does not recompute them, but every token costs memory. For Gemma 2 27B that is about 0.72MB per token, so an 80GB A100 holds only about 114k tokens. Hung-yi Lee then walks through ways to shrink it: let queries share keys and values (MQA, GQA), compress keys and values into one vector without ever decompressing (MLA), limit the attention span (Sliding Window, StreamingLLM), and drop keys and values nobody attends to (Scissorhands, H2O). He ends with cross-conversation prompt caching: it only hits when the prefix is identical, so a system prompt should put stable content first.
Looplane uses a static model-role catalog and retries or falls back only after retryable provider errors. Cache data is a provider hint plus trace, while cost is a static-table estimate; neither is live routing intelligence or a bill.
Claude Code's prompt caching works by exact prefix matching: a cache read bills at roughly 10% of the standard input rate, but switching models, changing effort, enabling fast mode, toggling MCP servers, or denying an entire tool forces the next turn to reprocess everything. The TTL defaults to five minutes; the main conversation and a few helper requests on a subscription get one hour.
Hermes supports 40+ providers, and the consumer-subscription OAuth paths are where billing surprises live: Anthropic OAuth only spends Claude Max extra-usage credits, and Claude Pro can't use it at all. Auxiliary tasks default to `provider: auto`, meaning your expensive main model does compression and vision grunt work. The fallback chain is a one-shot switch per session, not continuous retry.
A detailed look at OpenAI's Codex agent loop design: how prompts are constructed, how multi-turn conversations are managed, how prompt caching prevents cost explosions, and how context window auto-compaction works.
OpenClaw's failover runs in two stages: rotate auth profiles within the provider, then fall back to another model. But what really governs behavior is who chose the model — a model you picked yourself with /model is strict, and its failure is reported rather than answered by some other model.