Is It Worth Re-Explaining Someone Else's Course? The Research Debt Argument, Open Textbooks, Switching Textbooks, Minimal Manuals, and a Four-Way Split of Documentation

🌏 中文版

This is Part 27 of the "Writing Articles That Teach" series, in its last section, "how to know it works, and how to make it". Earlier parts covered what the signals in an author's hands actually measure, and what large-scale experiments and rubrics can tell us. This part steps back to a more basic question: should this kind of article be written at all?

The question comes from this site's course guides. Put plainly, what these articles do is re-explain someone else's course. The seven-week guide to Harvard CS50 AI writes up each week's lectures and projects in Chinese, based on the English subtitles of the lecture recordings and the course website. The lecture guides for Stanford CS229 are based on the subtitles and lecture notes of the public Spring 2026 recordings. The original courses are themselves public: the recordings, lecture notes, and assignment specifications are all online. So every article runs into the same question: why wouldn't readers just watch the original course? Is it worth the effort to explain it again?

I split this question into three parts. First, what is the argument that explaining things clearly is worth extra effort? Second, how much do learning outcomes change when the same course uses different materials? The closest data here are comparisons of open and commercial textbooks, and comparisons of different textbooks on the same subject. Third, what should a rewrite look like? This covers experiments that rewrote existing manuals, and the practice of sorting documentation into types. The last section covers what comes after publication: how much effort it takes to maintain a teaching article once it is online.

How sources were chosen, and their biases. I scanned the literature first, then picked sources to read closely. The criteria: every conclusion that uses a number needs a source I could read from start to finish, and the set had to include studies with opposite or null results. I ended up reading seven: a position paper (Research Debt); a meta-analysis on open textbooks (pooling the results of many studies into one estimate) and a narrative review; a quasi-experiment comparing textbooks; a meta-analysis of minimal manuals; three pages of the Diátaxis website; and an empirical study of whether teaching articles can be executed. The biases are clear. All of it is English-language literature. The open-textbook and textbook-comparison studies were almost all done in US schools, with teachers and grades. Not one studies self-learners reading a guide on a web page. I read every item in the references from start to finish and kept an extraction record for each; studies I only know exist, without having read a legitimately public full text, are flagged as such in the text, and I don't cite their results.

The short version

g and d are both standardized mean differences: around 0.2 counts as small, around 0.5 as medium, 0.8 and above as large. The table does not say that re-explaining is useless. The open-textbook and textbook-comparison studies all compare switching to a different material on the same subject, not explaining the same content again, more clearly. The latter is exactly the part nobody has tested.

Research debt: the one-to-many argument

Chris Olah and Shan Carter published Research Debt on Distill in 2017. It is a position paper with no participants and no data. Its argument rests on metaphor, a formula placed in a footnote, and the authors' own experience.

The essay opens by likening the path to research-level understanding to climbing a mountain, and says that roads and stairs could in fact be built on the mountain; a footnote explains that the roads and stairs are truly outstanding tutorials, review articles, and textbooks. When nobody builds the road, everyone who comes later has to climb on their own. That is research debt. The essay's definition is "Research debt is the accumulation of missing interpretive labor." Interpretive labor refers to the trade-off between the effort of the person explaining and the effort of the person understanding: the more the explainer puts in, the less the understander has to. The core argument concerns one-to-many situations:

In these one-to-many cases, each member of the audience pays the cost of understanding, even though the cost of explaining stays the same.

In other words, an explanation is written once, and the more people read it, the more understanding cost it saves across all its readers. A footnote writes this claim as a complexity formula, and also converts the readership of one author's blog into time saved (the numbers are in the appendix). The coefficients in the formula have no estimated values, and the readership figure is the authors' own account, with no source.

The essay also lists four items of research debt: poor exposition, undigested ideas, bad abstractions and notation, and noise. Notation was covered in Part 4.

Before using this essay to support the claim that re-explaining a course is worthwhile, three things need to be stated.

First, it is about researchers in a research community understanding one another. The footnote's formula assumes N people understanding each other, not one author writing for N readers. The essay does not discuss course materials or general learners. Using it to support course guides is my extension.

Second, it puts a condition on who does the re-explaining. The essay says research debt can't be solved by one person writing a textbook, because one person's energy isn't enough to polish every idea from scratch. Nor can it be outsourced:

We can’t outsource distillation to less skilled non-experts: refining and explaining ideas requires creativity and deep understanding, just as much as novel research.

The paragraph before that also says that good explanation often requires transforming the idea itself, and that this transformation can be as hard as discovering the idea in the first place. So the essay is not arguing for more explanations. It argues for explanations written with great effort by people who truly understand.

Third, its answer to why so few people do this work is the authors' opinion. The authors think the main cause is not bad incentives (a footnote says this is their experience, and other fields may differ) but a lack of support: people who do explanation lack careers, places to learn, and role models, and the work isn't treated as a real research contribution.

Many of this site's guides are drafted by an AI agent from lecture notes and subtitles, then reviewed by another agent (Part 22 mentioned the drafting). By the standard of the passage above, no study has tested whether this process counts as outsourcing to less skilled non-experts. Later parts will cover the evidence on AI-written teaching materials, and which errors this site's review process actually caught.

Open textbooks: no reliable difference in learning, fewer withdrawals

The largest body of data closest to how much learning changes when you switch materials is research on open textbooks. An open textbook (a kind of open educational resource, or OER) is openly licensed and free for students; studies compare it with commercial textbooks that students have to buy.

The meta-analysis: an average of zero

Virginia Clinton and Shafiq Khan's 2019 meta-analysis preregistered its analysis plan and included 22 studies, all in university and college courses, mainly in the US and Canada. The authors say in the introduction that textbooks can almost never be randomly assigned, so nearly all the included studies are quasi-experiments: classes using open textbooks and classes using commercial ones often had different instructors, different semesters, and different exams. Learning performance was mostly the course grade or whether a student got a C or better; only 8 studies used the same measure for both groups.

The main result is that learning performance showed no reliable difference, g = 0.01. The authors also ran an equivalence test and concluded that the effect was statistically equivalent to zero within ±0.2 standard deviations. Results varied a lot across studies, with individual studies ranging from clearly worse to clearly better. The authors tried to explain these differences with three variables: whether prior achievement was controlled, whether the instructor was the same, and whether the measure was the same. For two of them, the degrees of freedom were too low, and the authors said the results could not be trusted and were inconclusive. For the third, they could not tell whether there truly was no difference or the test lacked power. So the abstract's statement that no significant moderators were found means none were detected, not tested and found absent.

On withdrawals, courses using open textbooks had fewer withdrawals, with an odds ratio of 0.71 (11 studies). This result is weaker than the one for learning. For 6 of the 11 studies, the withdrawal figures were obtained privately from the original authors, so the authors did not check for publication bias; there were also too few studies for a moderator analysis, and one study went significantly in the opposite direction. The authors themselves urge caution. The result can only be written as courses using open textbooks had fewer withdrawals, not as open textbooks made students withdraw less.

The two results need to be read together. In their limitations, the authors point out that students who withdraw usually have weaker academic backgrounds. If the two groups' withdrawal rates differ, so does the makeup of the students who stay and finish the course. The g = 0.01 for learning was computed among students who finished, and the students who stayed in the two groups may not be alike.

When discussing why learning showed no difference, the authors list several possibilities. The last is that the type of textbook may inherently have little effect on learning ("the type of textbook simply has little influence on student learning"). This is speculation, not a result of the meta-analysis. As support, the authors cite another experiment comparing textbooks from different publishers (Durwin and Sherman 2008) and two studies in which all students had access to commercial textbooks.

The narrative review: small but positive

John Hilton III's 2020 review covers OER research published from 2015 to 2018, including 16 studies that compared learning outcomes. It shares 12 studies with the meta-analysis above.

The method differs. Hilton did not compute effect sizes, instead summarizing each study and counting how many leaned which way. At the end of the table, Hilton counts 9 studies favoring open materials and 1 favoring commercial textbooks; the rest found no difference, had mixed results, or did not discuss significance. Combining studies from 2002 to 2018, Hilton concludes that open materials do not harm learning and that their effect is small but positive. Hilton also says this is a qualitative review and that a quantitative meta-analysis is still needed.

In the study-by-study details, several points say more about switching materials than about open or not:

• Which material the test is aligned with decides who wins. Study 2 of Gurung 2017 favored commercial textbooks overall, but when the questions came only from the open textbook's test bank, no difference was detected.
• Accompanying practice may matter more than the text. In Chiorescu 2017, the instructor used open materials and then switched back to a commercial textbook, and grades and withdrawal rates returned to their earlier levels. The reason for switching back was that the online math practice system paired with the open materials was worse.
• Under the average are groups moving in opposite directions. In Colvard 2018, the rate of D, F, or withdrawal dropped sharply for part-time students and rose slightly for full-time students.
• Switching textbooks often coincides with other changes. In Hendricks 2017, teaching methods were heavily revised at the same time as the textbook changed.

These are all single studies as reported by Hilton.

In the conflict-of-interest statement, Hilton discloses research funding from the William and Flora Hewlett Foundation and says that funding was not directly related to this review. Some of the 16 studies also had Hilton as an author.

Readers say open materials are as good or better

Hilton also compiled 20 perception studies. In the studies that asked students or instructors who had used both kinds of materials to compare them directly, every one found a clear majority saying the open materials were as good or better. But in the 5 studies where each group rated its own book, 2 rated the commercial textbook higher, 2 rated the open materials higher, and 1 found them similar.

Hilton has three reservations about these results. First, the main pattern is that students dislike paying for textbooks and so prefer the free option; many instructors also notice this preference, which may affect their own ratings of open materials. Second, in the largest student survey, the share of students who rated the open materials better was almost identical to the share who rated the course's teaching better, so goodwill toward the course as a whole may have carried over to the materials. Third, Hilton writes explicitly that "student preference does not necessarily equal better learning," and reports another study in which no correlation was detected between students' quiz scores and their ratings of the textbook.

Earlier parts discussed, from the side of the signals an author has, why readers liking something doesn't mean they learned from it. For how accurately readers judge whether they understood, see Part 15.

What these studies compare

The open-textbook studies compare license and price. The two books are different books by different authors, and the difference mixes price, ease of access, content, and instructor. They do not test explaining the same content again, more clearly. The conclusion useful to this site goes only this far: in university courses, switching to a free open textbook makes, on average, no reliable difference in learning performance.

Switching to another textbook on the same subject

What if price is taken out of the picture, and only different textbooks for the same subject are compared?

Cory Koedel and colleagues' 2017 study used the four elementary math textbooks adopted in California's 2008 and 2009 adoption round. All four had passed state approval. The unit of analysis is the school. The authors hand-copied which textbook each school used from its public school report card, compared school-average grade 3 state math test scores, and in the end analyzed 1,878 schools. Textbooks were not randomly assigned, so the authors used propensity score matching to make schools that adopted different textbooks as similar as possible on 22 conditions, such as pre-adoption scores, student composition, and district size.

The main comparison is one textbook, California Math, against the other three combined. Schools using California Math scored higher. Of three estimation methods, the two that reached statistical significance estimated 0.05 to 0.065 student-level standard deviations. The authors converted this into school rankings: roughly, it moves a school ranked in the middle up to the 54th to 56th percentile. The matching estimate the authors preferred on conceptual grounds had similar point estimates (0.06 to 0.08) but much larger standard errors, and none of the four years' estimates was significant.

Several conditions go with this result:

• Only one textbook looked better; no difference was detected among the other three. The authors first ran six pairwise comparisons. Only 1 of the 24 estimates was significant, and power was insufficient. The main comparison was redesigned as one book against the other three combined after the authors had seen these results. The claim that the other three are about the same rests on a set of non-significant point estimates with large standard errors; the authors' wording is "point toward."
• Longer use was not better. Cohorts that used the book for three years showed no larger effect than those that used it for one year. The authors list several possible reasons, all speculative.
• The authors can't say what makes this book better. Public data on the four books' features are scarce. The authors say that for linking textbook features to effects, there are currently too many explanations and too few estimates.
• The authors say this gap is at the low end of earlier similar studies. The earlier studies reported by Koedel and colleagues range from 0.08 to 0.17.

The authors stress that the marginal cost of choosing one textbook over another is close to zero, while the choice affects every student who uses the book. Note that the authors admit in a footnote that they don't know the four books' actual prices; close to zero is inferred from average prices in other studies.

Another study is Blazar and colleagues' 2019 textbook comparison (Harvard CEPR's Learning by the Book). The study exists, but I have read only the project page, not the report itself, so I don't cite its results and can't set it side by side with Koedel and colleagues.

In its introduction, Hilton's review reports two earlier syntheses. Slavin and Lake's 2008 synthesis of math curricula concluded that improvements in teaching matter more than the choice of curriculum. In 2004, the US National Research Council reviewed the research on 19 K-12 math curricula and could not determine which was most effective.

Taken together, what can be said is this: among approved textbooks for the same subject, switching to another one has a very small effect on average scores, and the research can't say which features make the difference. This is still not re-explaining. The four books were written from scratch by four separate teams; they are not rewrites of the same content.

Rewriting existing materials: minimal manuals

The research that really does compare taking existing material, rewriting it, and setting it against the original is research on computer manuals from the 1980s.

The approach was to redesign a manual originally organized around system functions into a minimal manual. Representative studies are John Carroll and colleagues' 1987 The Minimal Manual, and Ard Lazonder and Hans van der Meij's 1993 The minimal manual: is less really more?. Both studies exist, but I could not get a legitimately public full text of either.

What I could read in full is the meta-analysis by Paul Ginns, Nina Hollender, and Peter Reimann. It is a document deposited in ERIC, with no journal name and no DOI, and nothing in it shows that it went through journal review. The document prints no year; its search ran up to March 2006. According to this document, minimalist design has four principles: anchor the material in the tasks users actually need to do, get people acting from the start, support recognizing and recovering from errors, and cut unnecessary explanation.

The main analysis compares minimalist materials with the traditional approach organized around system functions: 13 effect sizes in total, 288 people, mean d = 1.12, a large effect. All 13 effect sizes are positive.

This number has to be read together with the following points:

• 3 of the 13 are not comparisons of manuals. Those 3 compare self-study with software features unlocked step by step against training led by a professional instructor. The document doesn't report separately what the figure is for the 10 minimal-manual effect sizes alone.
• The studies are old and the domain is narrow. The original studies date from 1985 to 1994, and 10 of the 13 effect sizes involve word-processing software. Most of what they measured is performance in following procedures (completing test tasks, number of errors), not conceptual understanding.
• Samples are small, and publication bias was not checked. 288 people across 13 comparisons works out, by my calculation, to about 22 per comparison on average; the authors say some studies had fewer than 10 per group. The document contains no check for publication bias at all, and all-positive effect sizes with small samples are exactly the situation that most needs one.
• Many studies come from the teams that proposed the method. In the acknowledgments, the authors thank Carroll, Siegfried Greif, and van der Meij for help finding studies, and many studies in the table come from these researchers' teams. The document does not treat this as a limitation.

The document also runs two analyses of single principles, each with only 3 studies: strengthening error recognition and recovery, d = 0.59; cutting unnecessary explanation, d = 0.89 (one of these studies measured completion time). Seeing that the latter effect was close to the main analysis, the authors speculate that cutting redundancy may be the main source of a minimalist rewrite's effect, and label this themselves as speculation that would need factorial experiments to test.

The line most directly relevant to whether rewriting is worth it is the authors' reminder, in the discussion, to take seriously the large effort a redesign takes ("it is vital to recognise the considerable effort that should go into such redesigns"). The authors go on to say that minimalist materials should be developed iteratively, with usability testing with real users at every stage. They attribute the largest effect (d = 2.34) to the effort put into designing that manual from a detailed task analysis. So these studies support rewrites work when they go through task analysis and user testing, not rewriting works. Whether usability testing itself contributes was not compared in this meta-analysis.

Should documentation be split into types: Diátaxis

When re-explaining a course, you also have to decide what kind of document to write: an explanation that follows the lectures, steps that take readers through a project, or a reference to look things up in. One split commonly used in technical documentation is Diátaxis, which divides documentation into four types: tutorials, how-to guides, reference, and explanation. Part 9 already summarized what it rests on. Here I add another page and use it to check this site's guides.

Its basis is a conceptual derivation, not data. The Foundations page derives it from two binary dimensions: a craft involves both action and cognition, and it has to be both acquired and applied. Crossing the two dimensions gives four cells, and from this the page says documentation comes in "four and only four" types. The page asserts, without argument, that these two dimensions are the right ones and that no others are needed. The page discounts its own case at the outset: after saying the scheme is useful and feels right, it continues:

However, that’s not enough to be confident in Diátaxis as a theory of documentation.

The home page says it is "proven in practice," adopted by hundreds of documentation projects, and includes three testimonials from adopters. None of the three pages I read (the home page, Foundations, and How to use) cites any research; I have not read the rest of the site. The site also says the people it serves are practitioners in a domain of skill, for example people using a programming language.

Its way of working opposes building the structure first. The How to use page says the scheme is a guide, not a plan. Don't build four empty sections first and then fill them in, and don't tear a messy set of documents down and start over ("If you’re tidying up a huge mess, the temptation is to tear it all down and start again. Again, avoid it.").

What it recommends instead is to pick one small piece at a time and ask four questions: what kind of user need does this piece serve, how well does it serve it, what could be added, removed, or moved to serve it better, and do its language and logic fit what this type of document requires. Then make just one change that improves it right away, and publish once that's done. Reading only the first two pages makes it easy to take Diátaxis to mean split your documentation into four zones, which is exactly what the third page tells you not to do.

After publication: maintenance

The one-to-many argument carries a hidden assumption: write an explanation once and its cost is paid. For teaching articles that contain commands and code, that assumption does not hold.

Samim Mirhosseini and Chris Parnin's 2020 study collected 663 Ubuntu system-configuration tutorials from three websites (Vultr, DigitalOcean, Linuxize) and fed every code block, unmodified, into a freshly created virtual machine as a shell command. After confirmation flags were automatically added to installation commands, 26% of code blocks executed successfully.

This number is not the success rate of readers following along:

• The unit is the code block, not the article. About a quarter of the sampled blocks were never meant to be executed as commands; they were file contents, command output, or text for display, and the machine ran them as commands anyway. I calculated this share from the paper's classification counts.
• Once an earlier step gets stuck, later blocks never get executed. Counting only the blocks that actually ran, the success rate is about 40%; I calculated this too.
• Real people answer prompts, save file contents to files, and fill in their own IP address. The closest thing to someone interpreting the text and then following it is the group where the researchers added manual annotations: a sample of 40 articles, in which about half of the blocks succeeded, but not one of the 40 articles ran from start to finish.

The authors sort the remaining failures into five categories, without giving the share of each:

• assuming the environment is already prepared (for example, not saying the package list must be updated first, or prerequisites that link out to several other tutorials);
• file-content blocks that don't say whether to replace the whole file or change only one part;
• missing steps or contradictions;
• failed external resources (download links gone, keys expired);
• needing resources the reader doesn't have (a domain name, a paid cloud service).

The study asked 6 people responsible for technical documentation at large companies to look at the results and give feedback. One said maintaining tutorials is inherently hard, there is currently no way to automate it, and every update means editing by hand. Another said automatically executing and testing tutorials is a hard problem. The study measured no maintenance costs and has no reader data. The authors themselves say the sample doesn't support statistical inference about failure proportions, and that with more work on the execution environment the numbers would be higher.

The study's subject is system-configuration tutorials from hosting providers, not course guides. Carried over to course guides, the cost of re-explaining a course with assignments and code isn't paid only once, at writing time. A revision of the original course, a package update, or a dataset URL going dead can each make the article decay without anyone touching it. For whether the code pasted into an article runs, and the actual state of the old project code in this site's CS50 AI guides, see Part 3. The actual maintenance problems of this site's guides will be covered in a later part.

Where the sources disagree

Is the effect of open textbooks zero, or small but positive? Clinton and Khan's meta-analysis pools the effect sizes to g = 0.01, judged equivalent to zero by an equivalence test. Hilton's review says the effect is small but positive, on the basis that 11 of 25 studies favored open materials and 1 favored commercial textbooks. The two share 12 studies but use different methods: counting how many studies were significant ignores each study's sample size and effect size, and Hilton also says a quantitative meta-analysis is needed. The same study often has different sample sizes in the two syntheses (an example is in the appendix), so their figures can't be mixed.

Does switching textbooks make a difference? Koedel and colleagues found one book did better, with no difference detected among the other three. Clinton and Khan speculate that the type of textbook may inherently matter little. The earlier syntheses Hilton reports concluded that improvements in teaching matter more than curriculum choice. The three measure different things: four approved textbooks in one state, open versus commercial textbooks in universities, and K-12 math curricula. I couldn't read Blazar and colleagues' report, so it can't be included.

Do readers think open materials are good? In the studies that asked readers to compare the two kinds of materials directly, every one found a clear majority saying open materials were as good or better; in the 5 where each group rated its own book, the result was 2 to 2 to 1. Lawrence and Lester 2018 found the opposite: a higher share of students using commercial textbooks were satisfied. In Gurung 2017, students using commercial textbooks gave higher ratings, with the difference lying in figures, photos, and visual appeal. These are all single studies as reported by Hilton.

Has Diátaxis been proven? The home page says it is "proven in practice," on the basis of the number of adopting projects and three testimonials. The Foundations page opens by saying that being useful and feeling right is not enough to give confidence in it as a theory of documentation, then fills the gap with a conceptual derivation. The two pages give different weight to useful in practice.

Is vague wording in a tutorial a defect, or meant for humans? Mirhosseini and Parnin list file-content blocks that don't say how to edit the file as one cause of failure, but the documentation leads who gave feedback disagreed. One said this vagueness is meant to guide human readers to the corresponding spot in the file and was never written for machines to interpret; another thought abbreviating with ellipses doesn't help readers. In other words, the machine couldn't run it does not always mean a person following it would fail.

What you can do

There is no direct research on this topic. The five items below are practices I derived from indirect evidence, each with its basis.

1. Before writing, write down what this article gives readers beyond watching the original course. Basis: in the open-textbook and textbook-switching research, switching to another material on the same subject makes, on average, a small difference or no reliable difference. So the value of re-explaining won't come from being a different version in itself. It has to come from things the original doesn't have, such as the reader's language, prerequisites the course never taught, an example readers can work through, or self-checks with answers. The evidence for each of these is in earlier parts: prior knowledge in Part 12, language and translation in Part 24, examples in Part 1, self-checks in Part 9.
2. When rewriting, cut explanation readers won't use, and add how to notice and fix mistakes. The basis is two follow-up analyses in the minimal-manual meta-analysis, each with only 3 studies, all on software manuals from 1985 to 1995; cutting redundancy is the main cause is speculation that the original authors labeled as such. Weak evidence.
3. Find one or two real readers to try it, see where they get stuck, then revise. The basis is the minimal-manual researchers' recommendation: a rewrite should iterate, with testing by real users in every round. This is the authors' recommendation; the meta-analysis did not compare doing user testing with not doing it.
4. When revising an old article, change one piece at a time; don't tear it down and start over. The basis is the working advice on the Diátaxis How to use page, with no research behind it. Ask yourself four questions: what need does this piece serve, how well does it serve it, what could be added or removed to make it better, and do its tone and logic fit this type of document.
5. Count maintenance as a cost. The basis is Mirhosseini and Parnin's descriptive data and the feedback from documentation leads. For articles with commands and code, state the environment and versions, and periodically rerun the article from the beginning in a clean environment.

Checking this site's practice

The writing spec for this site's course guides is .agents/skills/post/references/course-guide-contract.md. Below is my judgment from checking this site against these studies. Five points relate to this part:

Taking CS50 AI Week 6 as an example: the first half, the lecture guide, follows the recording section by section, explaining syntax, n-grams, and word vectors through to the Transformer. It is closest to Diátaxis's explanation, but each section ends with a self-check question with a collapsed answer, and the article ends with a learning checklist; that layer leans toward tutorial. The second half covers two projects: the task, the code the course provides, the solution approach, pseudocode, common mistakes, and how to check your work. Readers complete an assigned task while they are learning, so this part has the character of both tutorial and how-to guide. The full reference implementation is collapsed at the end of each project for readers who are stuck, which is closer to reference. The series also has a cross-week review and a course retrospective, which roughly count as explanation. Whether this mix should be split up, I would decide with Diátaxis's own method: ask, piece by piece, what need each part serves, instead of first breaking the structure into four zones.

One more point bears directly on one-to-many. The value of that argument depends on how many people read, and whether reading actually saves them the effort of understanding. The production site loads at least Cloudflare Web Analytics, and its traffic statistics are aggregate (see Part 19); they can't measure whether an individual reader learned anything. So how much effort these guides save their readers is a question this site currently can't answer.

Three things you can do tonight

1. Pick a course guide you've written and add a sentence at the start saying what this article gives the reader beyond watching the original course. If you can't write that sentence, first find the hardest part of the original course and add an example readers can work through, or a self-check question with an answer.
2. Pick one subsection of the guide, run through Diátaxis's four questions, and make just one change. Most often that means cutting a passage of explanation readers won't use, or moving a stretch of explanation out of the middle of procedural steps.
3. Open a clean environment and follow an article that contains commands from start to finish. Note the first place you can't continue, fix it, and state the environment and versions in the article.

Limits of this evidence

• No study has directly tested re-explaining a course. The open-textbook studies compare two books with different licenses and prices. The textbook-comparison study compares books written separately by four teams. The minimal-manual studies mostly compare two manuals for the same software (3 of the 13 effect sizes are not manuals). The last is the closest, but it concerns software manuals from 1985 to 1995, not course guides.
• The settings are far from self-learning readers. The open-textbook and textbook studies were all done in schools, with teachers, exams, and grades; learning performance was mostly course grades or state tests. No study has measured readers who come on their own to read a guide on a web page.
• Two of the sources are arguments, not data. Research Debt and Diátaxis are both their writers' own arguments and experience, with no controlled studies. I found no study evaluating Diátaxis; not finding one doesn't mean none exists.
• The minimal-manual meta-analysis is weak: no journal review, no check for publication bias, all-positive effect sizes, small samples, and studies mostly from the proposers' own teams.
• The executability study used machine execution, not real people following along; its unit is the code block; it covers only Ubuntu configuration tutorials from three websites; and the authors say the sample doesn't support statistical inference about failure proportions.
• Sampling bias: all English-language literature; OER research almost entirely from the US and Canada; research on Chinese open materials or translated materials was not covered this time. I read only three pages of Diátaxis.
• My inferences appear in seven places: extending the research debt argument to course guides; applying the point about not outsourcing to non-experts to AI-agent drafting; that the value of re-explaining has to come from what the original lacks; that re-explaining a course with code means paying ongoing maintenance costs; which Diátaxis types the CS50 AI weekly guides map to; that Diátaxis's own method should decide whether to split them; and the three things to do tonight. The text doesn't mark each one; they are listed here.
• Known to exist, but no legitimately public full text read: Blazar and colleagues' (2019) textbook comparison (only the project page read), the original minimal-manual experiments of Carroll et al. (1987), and the redo by Lazonder and van der Meij (1993). The text doesn't cite their results or list them in the references; the effect sizes for these two minimal-manual studies in the appendix are the numbers listed in the table of Ginns and colleagues' meta-analysis.
• Secondary citations: the three studies, including Durwin and Sherman (2008), reported by Clinton and Khan; Slavin and Lake (2008), the US National Research Council (2004), and Landrum et al. (2012), reported by Hilton, along with the original studies summarized one by one in that review; the earlier textbook studies reported by Koedel and colleagues; and Merritt (1997), reported by Ginns and colleagues. I have read none of the originals.

Appendix: design and numbers for each study

The main text keeps only one or two numbers per item; the rest are here.

Olah and Carter (2017), Distill. Position paper, no data. The page lists four items of research debt: Poor Exposition, Undigested Ideas, Bad abstractions and notation, and Noise. The original wording is "several forms"; it doesn't say there are only four. The one-to-many argument takes up only two qualitative sentences in the main text. The footnote's formula: when N people understand each other, each writing one explanation is O(1), and understanding the other N−1 people's ideas is O(N), so the total cost looks like O(a + bN), where a and b are trade-off coefficients for the explaining side and the understanding side, with no estimated values. Another footnote says Christopher's blog posts are read by more than 100,000 people on average, so saving each reader one second saves humanity 30 hours (100,000 seconds is about 27.8 hours; the original gives an approximation). This is the authors' own account, with no source. The essay is about understanding within the research community, and the authors also say the Distill ecosystem "is just a start."

Clinton and Khan (2019), AERA Open 5(3). Preregistered. Five databases searched in August 2018; 461 deduplicated records screened by the first author alone, supplemented through mailing lists, emails to authors, and other channels; in the end 22 reports covering 23 independent studies. The learning-outcome analysis covers 22 studies and 26 effect sizes, using robust variance estimation. Learning performance g = 0.01 (SE 0.08, 95% CI −0.16 to 0.19, p = .87), I² = 98.17; equivalence test with bounds of d = 0.2, Z = −2.25, p = .01. The appendix code also ran bounds of ±0.1, which the text does not report. I computed it myself from the values in the code: with ±0.1 the test would not be significant, so equivalent to zero holds only under the ±0.2 bounds. The funnel plot and Egger's test show no sign of publication bias. Individual studies' g ranged from −0.87 (Gurung 2017, Study 1) to 1.104 (Grewe and Davis 2017); of the 22, 11 were positive, 1 zero, and 10 negative (I counted from Table 3). Same instructor: 12; controlled for prior achievement: 6; same measure: 8. Moderator analyses: controlled prior achievement df = 3.54 and same measure df = 3.34, both judged unreliable by the authors; same instructor p = .91, which the authors say can't distinguish no difference from insufficient power. Withdrawal: OR = 0.71 (95% CI 0.56–0.90, p = .005), 11 studies, 78,593 people, I² = 83.09. Individually, 3 ORs were < 1 and significant, 1 was > 1 and significant (Hilton 2013; its original authors say the math placement policy changed at the same time), and the other 7 were individually non-significant; for 6 of the 11, the authors requested the figures from the original authors. The abstract gives a total N = 100,012, but summing the two group columns in Table 3 gives 109,192.5, which doesn't match the total at the bottom of the table. Limitations: higher education only, geography mainly the US and Canada, abstract screening by one person, reading medium (print or screen) not controlled, and differences in withdrawal may change the makeup of those who finished.

Hilton (2020), ETR&D 68. Narrative review, single author. Inclusion criteria: OER as the primary learning resource of a university course, some comparison with commercial textbooks, at least 50 people, published between September 2015 (the methods section says October) and December 2018. The search started from the 16 studies in the author's 2016 review, looking for articles citing them in Google Scholar, plus a search of ProQuest dissertations and inquiries to people who had done related research; screening was done by the author alone. 16 efficacy studies with 114,419 students, of whom 27,710 used open materials; 4 in the table controlled for both instructor and student differences. 20 perception studies with 10,807 students and 379 instructors. Across 2002 to 2018, 25 efficacy studies in total: 3 didn't report significance, 10 found no difference or mixed results, 11 favored open materials, 1 favored commercial textbooks. The conclusion says more than 95% of the studies show that open materials do not make learning worse. The same study can have different sample sizes in the two syntheses; for example, Winitzky-Stephens and Pickavance 2017 is 7,588 vs 26,538 in Hilton, and 329 vs 1,120 in Clinton and Khan. Gurung 2017 Study 1: controlling for ACT scores, the open-materials group was 1.2 points (13%) lower. Colvard 2018: the rate of D, F, or withdrawal rose from 6.3% to 7.4% for full-time students and fell from 34.3% to 24.2% for part-time students. Lawrence and Lester 2018 perceptions: 74% of commercial-textbook users were satisfied, versus 57% of open-material users. Griffiths 2018 student survey of 2,350 people: 29% rated the open materials' quality slightly higher and 30% much higher, and the distribution for teaching quality was almost identical. The direct-comparison perception studies all surveyed people who were already using open materials, with no control group. Several internal numbers don't match: the number of studies covering both efficacy and perception is given as 7 and as 6; the total number of efficacy studies as 25 and as 22; and the total number of students in the perception studies as 13,302 and as 13,173.

Koedel, Li, Polikoff, Hardaway, Wrabel (2017), AERA Open 3(1). The four elementary math textbooks adopted in California in 2008 and 2009: enVision Math California, California Math, California Mathematics: Concepts, Skills, and Problem Solving, and California HSP Math. Starting from 5,494 schools, the authors excluded schools with missing textbook information, mismatched adoption years, different books in grades 1 to 3, or a book that was not one of the four, as well as the two large districts of Los Angeles and Long Beach, which were dropped after the analysis began. That left 1,878 schools in 309 districts. School-average grade 3 state math scores were multiplied by 0.45 to convert to student-level standard deviations; bootstrap resampling by district, 250 replications. Main results (California Math vs the other three combined, years 1 to 4 after adoption): kernel matching 0.063, 0.083, 0.061, 0.070, none significant; OLS restricted to common support 0.050, 0.064, 0.049, 0.058; matching after residualizing on out-of-sample schools 0.050, 0.065, 0.052, 0.060. The latter two were significant at p ≤ .05 in every year. Falsification tests: the placebo effects for pre-adoption cohorts and the effects of the math textbooks on English scores were all non-significant; the authors say this can't prove the conditional independence assumption holds, only look for signs that it is violated. The propensity score model explains only about 12% of the variation in adoption; after matching, none of the 22 covariates was imbalanced at the 5% level. Of the 24 estimates in the six pairwise comparisons, only 1 was significant. Grade 4 estimates are all positive but smaller and mostly non-significant; grade 5 estimates are larger. Split by socioeconomic disadvantage, the effect comes mainly from disadvantaged students; the authors say not to over-interpret this. Cost: citing other research, the authors put the average for elementary math textbooks at US$34 per student, about 0.32% of per-student spending; they don't know the prices of these four books. No preregistration; no data on how the textbooks were actually used.

Ginns, Hollender, Reimann (ERIC ED491708, ca. 2006). 13 effect sizes, 288 people, drawn from 5 journal articles, 3 conference papers, 1 technical report, and 1 doctoral dissertation; all randomly assigned. 10 are minimal manuals, and 3 are systems with features unlocked step by step (compared against training by a professional instructor). Random effects, d = 1.12 (95% CI 0.83–1.41), homogeneity test Q = 14.56, df = 12, p = 0.266. The studies in the table: Gong and Elkerton 1990 2.34, Lohmann 1994 Experiment 1 2.26, Carroll et al. 1987 Experiment 1 1.63, Gong 1990 1.54, Van der Meij and Lazonder 1993 1.26, Wendel and Frese 1987 1.14, Lazonder and van der Meij 1993 1.01, Lohmann 1994 Experiment 2 0.95, Carroll et al. 1987 Experiment 2 0.80, Oatley et al. 1990 0.77, Carroll et al. 1985 0.68, Ramsay and Oatley 1992 0.64, Lohmann 1994 Experiment 3 0.61. Error recognition and recovery: 3 studies, 118 people, d = 0.59 (0.22–0.96). Cutting unnecessary explanation: 3 studies, 120 people, d = 0.89 (0.36–1.42), where Black et al. 1987 measured time to complete the test exercises. For the other two principles (anchoring in tasks, action orientation), multiple experiments could not be found. The discussion reports an anecdote from Merritt 1997: development cost US$500,000, 5,000 copies sold in the first year, and a profit of US$19 million. The sentence defining the effect size refers to "integrated and non-integrated conditions," which doesn't match the condition names in this study.

Diátaxis (diataxis.fr): three pages (home, Foundations, How to use), current site pages with no publication date. Foundations' mapping: learning to tutorials (acquisition, action), goals to how-to guides (application, action), information to reference (application, cognition), understanding to explanation (acquisition, cognition). The home page's three testimonials come from staff working on the developer documentation of Vonage, Gatsby, and Cloudflare; all are subjective assessments with no before-and-after measurement. The How to use page says its way of working "runs counter to much of the accepted wisdom in documentation," discouraging up-front planning in favor of small, immediate iterations; the mainstream practice it refers to is not cited.

Mirhosseini and Parnin (2020), ESEC/FSE 2020; I read the author's version. Of 780 crawled tutorials, after removing duplicates, discontinued distributions, and articles mainly operated through a graphical interface, 663 remained (Vultr 339, DigitalOcean 224, Linuxize 100), containing 14,876 code blocks. Executed verbatim: 1,935 succeeded (13%), 3,172 failed (21%), and 9,769 (66%) were never executed because of an earlier timeout; 75% of tutorials timed out at some step. After adding -y to apt commands: 3,916 succeeded (26%), 5,646 failed (38%), 5,314 (36%) were not executed, and 39% of tutorials timed out. The most common failure was command not found, at 2,393. Classification of a stratified sample of 40 articles with 787 blocks: commands 573, file contents 111, command output 93, display 10. Execution after manual annotation: 131 skipped; of the remaining 656, 343 succeeded (52.3%). Of those, 47 blocks were not run because they would break the test environment and were recorded directly as failures; excluding them gives about 56% (my calculation). None of the 40 timed out, and none ran all the way through. The paper doesn't report these 40 articles' success rate under verbatim execution, so the doubling claimed in the abstract compares the 40-article result with the 663-article result. At least one of the 6 people who gave feedback was already using the tool to fix their own tutorials. The paper gives no statistics on when the data were collected or when each tutorial was published.

References

I read each item from start to finish and kept an extraction record; parts I couldn't read are noted in parentheses.

• Olah & Carter (2017), Research Debt (Distill page; I did not look at the three illustrations on the page or the revision history).
• Clinton & Khan (2019), Efficacy of Open Textbook Adoption on Learning Performance and Course Withdrawal Rates: A Meta-Analysis (published version; the quality-coding table in the appendix was not checked against the page layout, and I did not look at the violin plot of withdrawal rates).
• Hilton (2020), Open educational resources, student efficacy, and user perceptions: a synthesis of research published between 2015 and 2018 (the publisher's open-access published version; the two tables were read as converted text, not checked against the layout; I could not read the values in the bar chart).
• Koedel, Li, Polikoff, Hardaway & Wrabel (2017), Mathematics Curriculum Effects on Student Achievement in California (published version; the grade 4 and 5 results appear only in figures, with no table in the text).
• Ginns, Hollender & Reimann, Meta-Analysis of the Minimalist Training Model (ERIC ED491708) (a document deposited in ERIC, with no sign of journal review; the document prints no year).
• Diátaxis home page, Foundations, Diátaxis as a guide to work (retrieved 2026-10-13; I did not read the rest of the site or look at the figures on the pages).
• Mirhosseini & Parnin (2020), Docable: Evaluating the Executability of Software Tutorials (I read the author's version on the author's website, not compared word for word with the published version).
• Articles on this site: Writing the Examples and Openings of a Teaching Article, How Far Should the Code in a Teaching Article Go, Handling Terms, Symbols, and Formulas in a Teaching Article, Writing Questions and Wording in a Teaching Article, Writing a Teaching Article for Readers at Different Levels, Do Readers Know Whether They Understood?, Writing for Others, Learning for Yourself, Non-Native and Mobile Readers of a Teaching Article, Harvard CS50 AI course overview, CS50 AI Week 6, CS50 AI cross-week review (1), Stanford CS229 Lecture 1.