Skip to content

All posts

Page 7 of 128

What's Different When AI Reads Your Knowledge Base: Write the Test Questions First

A wiki that people understand can still trip up an AI: what it sees is a few chunks cut out of the pages, without the navigation, heading hierarchy or surrounding text. So before building RAG, write a set of test questions: each with its answer and its source passage, plus a few deliberately unanswerable ones. Azure's RAG guide does this in parallel with content analysis; one study found that evaluating only with AI-generated questions overstated retrieval performance by 29 to 37 percentage points, so mix in real users' questions. If your knowledge base is under about 500 pages, you may not need RAG at all: put the whole thing in the prompt.

Building a Knowledge Base: Four Questions for Personal Notes, Team Wikis and AI Knowledge Bases

Personal notes, a team wiki and a knowledge base for AI differ in who reads them: your future self, your colleagues, a model. All three have to answer what to keep, how to organize it, how it gets found and who maintains it. Research on personal collections finds that most 'finding' is re-finding things you have already seen; research on enterprise wikis finds that once the person who tidied or wrote the content leaves, it starts to decay.

Choosing a Note Tool: Obsidian, Notion, and Whether Your Data Comes With You

Before choosing a note tool, ask five things: can other programs open its files, do links survive renaming, can you search and filter, does it sync across devices, and does it work offline. Obsidian's notes are just Markdown files in a folder on your computer, but syntax such as block references can't be followed outside Obsidian, and on iPhone only two sync methods are officially recommended. Notion's databases are powerful, but an export takes only one view, callouts become HTML, and an export can't simply be uploaded back to restore your workspace. Before switching tools, export once and open the result in another program.

Where to Put Things: PARA, Johnny.Decimal, and Whether to Use Folders at All

For files people create themselves, research keeps finding that most retrieval happens by clicking through folders, and better search tools haven't changed that much; few people use tags, and those who do mostly apply just one. So notes are worth organizing, but each note needs only one main home, the structure shouldn't get too deep, and folders shouldn't get too full. PARA sorts things by how close they are to your current work, Johnny.Decimal allows at most ten per level and fixes each item's place with a number; the numbers in both are their authors' experience, not experimental findings.

Writing for Colleagues: How-to Templates, Decision Records, and Writing Less

Getting team documents to look alike and make sense takes two templates and one principle. The principle is fewer but fresh: Google's documentation guide says a small set of fresh, accurate docs beats a large pile in disrepair, and if something exists elsewhere, link to it. The how-to template runs prerequisites, steps, how to confirm it worked, and whom to ask when it fails; in one study, installation instructions rewritten to a template scored better on eight of ten aspects readers rated. Architecture decision records (ADRs) note why a decision was made; when one is reversed, you don't delete it, you mark it superseded and link to the new decision. Research also finds ADRs are often tried and dropped, and that the "considered options" section is the most problematic, often missing or misplaced, so keep the template short and keep that section.

Information Architecture: Organizing a Team Wiki by Department or by Task, and How to Test It

A common team-wiki argument is whether to organize by department or by task. The research is more cautious than you might expect: employees who know the org chart were actually faster on an intranet organized by department, and label quality mattered far more than the organizing scheme. A safer approach: give the main structure a scheme where every page has an owner, then add a few task-based entry pages that hold only links, and title pages with the words readers will search for. Before organizing, use card sorting to learn how readers think; afterwards, use tree testing to check whether they can find things.

Maintenance: Who Owns It, How Often to Check It, and When to Delete It

Documentation going out of date is normal, and it throws no errors: eight in ten of GitHub's most popular projects have at some point left references in their docs to code that had been deleted. There are four ways to maintain docs: give every page a responsible role, set verification expiry, change the docs in the same change as the process, and fix pages when they're used. Each one fails in its own way: the owner leaves, expiry notices get treated as spam, the rule exists but nobody follows it, and articles nobody uses never get checked. The deletion principle: delete what's certainly wrong first, and archive what's correct but rarely used. By the end you'll be able to draw up a maintenance table for your team: document type, responsible role, review cycle, and deletion conditions.

Findability: Using Search Logs to Find the Gaps, and the AI That Has Started Reading Team Docs

When colleagues can't find something, it usually isn't missing: of 50 zero-result queries at one university library, only one in ten was for something it didn't have. Sort the gaps in three to six months of search logs into can't be found, doesn't exist and different words, and fix titles and synonyms first; AGENTS.md files for AI need the same care, since one evaluation found they didn't raise success rates overall while raising costs by about a fifth.

Why Team Wikis Rot, and Which Kinds of Documents to Write

Research on rotting team wikis keeps finding three things: gaps and outdated pages are complained about more often than wrong ones; the people who write have no time or have left; and managers want something different from the people who actually write. Not every wiki rots: wikis hold up better when the writers are also the readers, or when the documents solve a clear problem. Diátaxis sorts documents into four kinds (tutorials, how-to guides, reference and explanation), and two questions tell you which kind a document should be, so you can see which kind your team is missing.

Maintenance: Keeping Notes Alive Past Three Months, and What to Hand an LLM

Schedule a 30-minute weekly review: empty the inbox, deal with finished projects, and pick one question from an MOC to think further about. Summarizing needs no schedule; add a layer only when you touch a note anyway. An LLM can find duplicates, suggest possibly related notes and flesh out key points you've written first; the sentence in your own words, deciding what matters and whether to delete are yours. In one 30-person study comparing three kinds of AI help, the group given complete AI notes learned the least and edited the least.

What to Keep: Collecting Is Not a Knowledge Base

To decide whether to keep something, ask three questions in order: can I name a concrete use for it, can I find it again later, can I write one sentence of my own about it. No use, don't keep it; if the original is findable again keep only a link, otherwise save a copy; only items with that sentence go into the knowledge base, the rest go on a to-read list. Empty the inbox every week. The reason behind it: every keep-or-skip decision risks two errors, keeping too much buries what matters and keeping too little means searching again, and loosening or tightening your threshold only trades one error for the other.

How to Link: Zettelkasten, Evergreen Notes and MOCs

After this post you'll be able to rewrite a reading summary as two or three notes that link to each other, and build a map note (MOC) once your notes pile up. The method comes down to four principles: one concept per note, organize by concept rather than by book, write in your own words, and state why you make each link. It goes back to Luhmann's slip box; research supports the idea that generating something yourself beats just reading it, but I found no credible controlled experiment on the Zettelkasten itself.

Cloudflare Model Family: Choosing Clef Decision Models and Clairic Extraction Models

Cloudflare's own models come in two classes: the Clef family returns option probabilities (clef 27B, clef-flash 9B, clef-omni 30B-A3B), while Clairic returns typed values and intervals. All four bill input tokens only ($0.038–$0.24 per million) and charge nothing for output; the three Clef models ship Apache-2.0 weights, and Clairic is still a hosted preview. They beat Jev on decision benchmarks and trail it on reasoning-heavy ones.

Choosing a Decision Model: Jev, Decider, d1-ultra, mammoth-1, and a Dozen More

Decision models replace an LLM's "generate some text and you write the parser" with "you list the options up front and the model returns a probability for each." After TypeSafe proposed System One on 2026-09-15, this category went from one vendor to more than a dozen: Jev 1.14, the OpenAI Decisions API, Perplexity Decider v1.2, Microsoft-Decision-1, Liquid d1-ultra, Base Labs mammoth-1, vLLM d3, StartLux, Fastino GLiDE, Celeris-1 Decision, Inception Mercury Decide, and Cloudflare Clef. The two leaderboards disagree almost completely — the community Jev Decision Index 0.4.1 crown goes to GLiDE V1.2 (71.5), while the independent Artificial Analysis Decision Index crown goes to d1-ultra (65.2) with Jev 1.14 at only 57.1. Decision models win on classification and tool selection, lose on knowledge and reasoning, and are broadly overconfident.

aideep-dive

How the Claude Code Team Does Code Review: Seven Layers, Risk Tiers and Where Humans Stay

Anthropic says Claude writes about 90% of Claude Code's code. This guide walks through their review flow: every PR passes seven layers, human approvals scale with risk from zero to two, and the goal is at most one human per change with no rise in reverts.

Seeing What an Agent Does: Drawing the Execution Trace from stream-json | Harvard CS2680 Assignment 1

CS2680 Assignment 1 has you build a web page that drives Claude Code: submit a task, watch live what the agent reads, what it runs, and how it turns out, then show how much it cost, how many seconds, and how many turns. The key is reading the event stream from `claude -p --output-format stream-json --verbose`: tool_use and tool_result pair up by the same id, and subagent events hang back under the step that dispatched them via parent_tool_use_id. This post breaks down the seven requirements, shows 40 lines of Python that build the events into a tree and tell success from failure, then covers submission, grading, and the places development most often gets stuck.

Building a Coding Agent Harness by Hand: Four Tools and a Loop | Harvard CS2680 Assignment 2

CS2680 Assignment 2 (Mads-Loop) has you write your own coding agent harness: with a fixed small model, qwen3.6-35b-a3b, it fixes real SWE-bench Pro issues inside Docker containers and logs every step in a required format. The assignment page says plainly not to trust the design it hands you, and some of the 10 tasks can't be solved. This post covers the spec, the places on the page worth doubting, the five places a harness most often breaks and how to handle each (with a snippet that has actually been run), how to build your own four-task eval set, and how to tell whether a task really is unsolvable. No official solution.

Saving Money and Time Without Losing Tasks: A Cost-Optimization Competition | Harvard CS2680 Assignment 3

CS2680 Assignment 3 (Mads-Opt) is a harness competition: on the same SWE-bench Pro tasks, who solves the most, spends the least per solved task, and waits the least per solved task. The three metrics aren't weighted into one score. Instead submissions are ranked in skyline (Pareto) layers: if nobody is at least as good as you on all three and better on at least one, you're rank 1. This post covers the container setup and hard limits, how ranking and scoring work (with a worked example), a first estimate of what one task costs using the model prices on the assignment page, and how Lecture 8's cost and speed levers map onto the assignment. No official solution and no specific strategies.

Modern AI Systems, From Using Agents to Optimizing Them | Harvard CS2680 Course Overview

Harvard CS2680 follows one coding agent and takes it apart from the top down. You start by watching Claude Code run, then write your own loop, pick tools and manage context, grow it into multiple agents, add memory, learn to evaluate it, and finally cut its cost and latency. In the second half of the term you go further down and serve the model yourself. This post covers what Part I's eight lectures and three assignments teach, works one example by hand to show why every extra agent step costs more, and ends with a prerequisite check and a reading order.

Agents Are a Systems Problem: From One Inference Call to a Chain of Tool Calls | Harvard CS2680 Lecture 1

CS2680 Lecture 1 (2026-09-02) starts with how AI systems differ from traditional systems: computation is split across CPUs and GPUs, floating-point math doesn't give the same answer every time, and state is large enough that it has to move between GPUs. It then compares an agent to a new kind of computer (the LLM is the CPU, the context window is memory, the harness is the operating system) and shows that one agent request becomes a long chain of inference → tool → inference. That makes tokens per second the wrong target; what matters is task success rate, end-to-end latency, and cost per success. This post works through two agents' numbers to show how those metrics are used.