What Research Measured on AI-Written Teaching Materials and AI Tutors: Content Error Rates, Human Screening, Student Ratings, and Tutoring Experiments
When GPT-3.5 wrote solution hints for algebra and statistics problems directly, both the answer and the steps were wrong for 32% of the problems; after a human screened every problem, no difference in learning was detected between those hints and hints written by human tutors. Randomized experiments on AI tutors point both ways: an unguarded chatbot made high school students do worse on an exam, while the positive results mostly came bundled with worked solutions written for each problem, a teacher in the room, or extra class time. None of the studies tested the same content again after a delay.