Skip to content
All tags

#ab-testing

4 posts
learningdeep-diveWriting Articles That Teach

Can Experiments Test How a Teaching Article Is Written? Evidence from Platform A/B Tests, Scale-Up Replications, Rubrics, and Design-Based Research

In a pooled analysis of 50 homework experiments on ASSISTments, hints, breaking a problem into smaller steps, and worked examples beat showing only the answer by d = 0.16–0.27, measured on performance during the homework itself. Interventions that worked in a few courses were scaled up to 247 MOOCs, and none of the main hypotheses replicated. Most course-quality rubrics don't publicly report reliability or validity; in one university's test, none of nine dimensions had a significant direct path to pass rates. The reviews of design-based research all look at how reports are written; none compares whether the materials it produces are more effective.

#learning-science#ab-testing#evaluation#mooc#self-learning#course-guide#blog
aideep-dive跟成熟 coding agent 學設計

Prompt Version Control: Changing One Word Can Drop an Eval from 5/5 to 0/5

Looplane's prompt is now `m3-exact-edit-v4`: the version persists into artifacts; core/tool/interaction/runtime/instructions/skills/workspace/memory are composed as stable or dynamic sections; and positive/negative examples cover replace_text, unified diffs, and direct replies. Unit tests pin the structure, while live-eval coverage still needs expansion.

#coding-agent#prompt-engineering#versioning#eval#ab-testing
aideep-dive

How to Rigorously Compare Before and After Agent Changes: From Golden Sets to Statistical Testing

Even with temperature=0, LLM outputs can still fluctuate by up to 15% in practice. To rigorously compare agent changes, you need a frozen golden set, at least 3 runs per query averaged out, LLM-as-judge blind evaluation (pairwise preference flip rate reaches 35%), and paired statistical tests -- not just running each version once and going by feel.

#evaluation#rag#llm-judge#ab-testing#ai-agent#llm
aiguideThe RAG Techniques Compendium

RAG A/B Testing: A Scientific Approach to Comparing Pipeline Configurations

"Adding a Cross-Encoder feels better" is not a scientific evaluation. A/B testing tells you whether a change actually works, how much it helps, and which query types benefit.

#rag#ab-testing#experimentation#metrics#pipeline