From Waiting for Instructions to Acting First: Proactive Agents and General User Models | Stanford CS329Z Lecture 18

🌏 中文版

Video status: Recordings require sign-in or course authorization. Source details

Picture two assistants. The first always waits for you to ask; if you say nothing, it does nothing. The second watches: when a friend's wedding invitation arrives, it looks up suit rentals and budgets and puts the results in front of you. The first is reactive; the second is a proactive agent, the subject of Lecture 18 (Nov 30, Proactive Agents).

The difference is not model size but who moves first. A reactive agent's loop starts from your instruction; a proactive agent's loop starts from its observations of you. Observation needs eyes, inference needs a user model, and acting needs rules for when to interrupt. The assigned reading is Shaikh et al.'s General User Models, with Next Action Predictors and OpenClaw as additional readings; this post digests all three. Paper video peer reviews are also due that day. As of 2026-10-29 this lecture has no public slides, so this post is a preview built from the official schedule and assigned readings, not a record of what was said in class. Lecture numbers after 11 follow schedule order; the course site currently lists only dates and topics.

Before proactive assistance: observation, inference, and intervention

The official schedule places Proactive Agents on November 30 and Frontiers & Open Problems on December 2. Final demonstrations belong to the December 7–11 examination period, with exact time and location still TBA. As of October 23, 2026, these classes have no public slide links. This guide separates GUM, next-action prediction, and open questions without treating future lectures or research demonstrations as validated products.

Observations do not establish intent

Repeatedly opening an English contract could mean translating, comparing versions, or waiting for a colleague. The GUM paper turns unstructured observations into revisable natural-language propositions. Such propositions are uncertain user models, rather than explicit authorization. Preserve evidence and correction paths so one mistaken inference does not influence assistance indefinitely.

Predicting the next action does not decide whether to help

Next Action Prediction studies subsequent actions in human-computer interaction. Predicting that a user may send an email and deciding to send it are separate layers. The first requires predictive evaluation; the second requires value, interruption, and authorization checks. In this original contract example, start with a dismissible summary suggestion. Sending messages, modifying documents, and making commitments require their own action boundaries.

Final evaluation must allow non-intervention

Create cases for useful and timely help, useful but excessive help, incorrect inference, and suggestions repeated after rejection. Check quality, trigger timing, and suppression separately. Treat waiting as a legitimate output so “proactive” does not become constant interruption. The Gumbo research demonstration offers a design example, rather than proof that every deployment is mature.

Check: next-action prediction is accurate, yet users disable the assistant. Why? Answer criteria: predictive accuracy covers behavior estimation. Interventions may still be excessive, low-value, or based on unacceptable observation and retention. Evaluate those dimensions separately.

Reading 1 | General User Models

Source: General User Models

TL;DR

GUM uses Propose, Retrieve, Revise, and Audit to maintain natural-language propositions and confidence for assistance such as Gumbo.

Editorial judgment

Domain background

Personal assistants need history without treating observations as permanent facts; propositions need uncertainty and revision.

Intermediate guide

Method. Propose propositions from observations, retrieve related memory, revise conflicts, audit storage, and let assistance query the resulting model.

GUM takes any unstructured observation, such as screenshots, as input. Its output is a set of natural-language propositions, each carrying a confidence score. Seeing a wedding invite, it writes down that the user is invited to a friend's wedding; seeing repeated draft edits followed by related-work reading, it writes down that the user is stuck on collaborator feedback. High confidence for the former, low for the latter, all labeled honestly.

The architecture is four modules. Propose turns observations into propositions, Retrieve pulls related old propositions in as context, Revise updates them as new evidence arrives, and Audit blocks anything the user would not want recorded. Screen understanding runs on Qwen 2.5 VL, proposition reasoning on Llama 3.3 70B, both on the open route. The paper deliberately picks open models so data never has to travel to a third-party platform. In the implementation Llama runs on private servers, and only the user can access the propositions.

The examples persuade. The first author's GUM found him a suit-rental spot for a wedding, automatically constrained by his budget. GUM can also complete a prompt's missing context — which section you are writing, which papers you have read — so a bare "help me with this section" just works. That is the same grounding Lecture 3's RAG chases, only the source changes from a retrieval index to traces of your computer use.

The GUM paper reports 76.15% mean proposition accuracy in its email-based evaluation. Propositions with confidence 10 were rated completely accurate within the evaluated sample; this does not guarantee future high-confidence propositions. A separate deployment with five participants over five days provides small-scale evidence about calibration and experience, rather than a universal effect.

Gumbo is the demo application built on GUM. It watches the screen, grows a GUM, and mines it for suggestions. Before interrupting you, it computes an expected utility: the probability the suggestion helps times its benefit, weighed against the cost of a wrong interruption. The math comes from Horvitz's mixed-initiative framework, and GUM supplies the off-the-shelf cost-benefit estimates it always lacked.

To stop suggestion floods, a rate limiter caps interruptions at one per minute. Suggestions that pass the bar get executed as far as possible — searching, running code, organizing files — before anything is shown. Thumbs up or down feed straight back into the GUM as new observations. The paper is honest about boundaries too: computer use stayed disabled during evaluation because it was too slow and buggy.

Next Action Prediction on the schedule points at something concrete. A follow-up by Shaikh et al. formalizes predicting the next action from multimodal computer-use traces — exactly the capability Gumbo's pipeline points at. See the Course Material box at the end for the full mini-review.

The Audit module filters observations by contextual integrity. Plainly put: would recording this information fit the setting it appeared in? The evaluation's verdict is honest: mostly compliant, but violations are bad when they happen. That is why Lecture 18 pairs privacy and trust with proactivity. When a reactive agent errs, you are at least present; when a proactive agent errs, you are often away.

The paper gives the privacy paradox its own section. The same participants wanted an assistant that remembers enough, yet flinched at seeing their habits written out as propositions. There is no technical fix here, only a tradeoff: the more it remembers, the more it helps — and the more it can get wrong.

This also echoes Lecture 1's systems view. None of GUM's four modules is the model itself; all are engineering around it: observing, auditing, retrieving, revising. Good results are still stacked up by systems — except this time the system is about you.

Evidence and result. The study demonstrates calibrated propositions and a proactive prototype while documenting privacy concerns and small-deployment limitations.

Deeper points

• Audit governs proposition inclusion while Revise updates existing propositions; privacy and consistency are separate responsibilities.
• Gumbo estimates suggestion utility before rate limiting; inferring a need and deciding to interrupt are distinct judgments.

Reviewer’s one-line verdict

Remembering more and helping better are separate claims to test.

Your take-away

Expose inferences for correction and deletion and configure intervention separately from retention.

Reading 2 | Learning Next Action Predictors

Source: Learning Next Action Predictors

TL;DR

LongNAP combines parametric learning and contextual history to predict actions from longitudinal smartphone traces with semantic evaluation.

Editorial judgment

Domain background

Longitudinal traces contain habits and current context; prediction must distinguish enduring preferences from the present screen.

Intermediate guide

Method. Convert traces into action sequences, predict from longitudinal and current context, and compare with observed subsequent actions.

Evidence and result. LongNAP outperforms the listed fine-tuning and prompt baselines on semantic next-action prediction, measuring prediction rather than task completion.

Deeper points

• Vision-language models produce many action labels, so labeling errors can affect training and evaluation.
• Parametric memory and contextual history retain different information; user splits affect generalization claims.

Reviewer’s one-line verdict

Accurate prediction does not create permission to intervene.

Your take-away

Evaluate prediction, usefulness, and interruption cost separately instead of replacing all three with similarity.

Reading 3 | OpenClaw

Source: OpenClaw

TL;DR

The project uses a local Gateway to connect models and channels; this guide treats it as an implementation comparison.

Editorial judgment

Domain background

Connecting an assistant to communication channels adds both information and externally consequential actions.

Intermediate guide

Method. A Gateway manages the assistant and enabled channels, connects configured models, and receives or presents assistance through those channels.

Evidence and result. Public code illustrates a multi-channel personal-assistant architecture; this guide cites no benchmark or privacy certification for it.

Deeper points

• A local Gateway and remote models have different data boundaries; local execution does not imply all data stays on-device.
• Channels add external input and outbound actions, requiring source-trust and sending-permission checks separately.

Reviewer’s one-line verdict

Understand channels and permissions before increasing proactivity.

Your take-away

List enabled channels, data, and actions and check permissions and disable controls individually.

Where it sits in the course

Lecture 18 moves the agent's starting point from instructions to observations. Lecture 19 closes the course with open problems in multimodal, web and computer-use agents, followed by the final Demo Day.

Course material for this lecture

• Official slides: not yet public (as of 2026-10-29)
• Assigned reading: Shaikh et al., "Creating General User Models from Computer Use" (digested here)
• Additional readings: Shaikh et al., "Learning Next Action Predictors from Human-Computer Interaction"; Steinberger, "OpenClaw" (both digested here)
• Schedule: CS329Z site

Course video sources

The Lecture Recordings section of the official Logistics page (read live on 2026-10-10) says recordings are available only by logging into the course Canvas site. I opened the course Canvas entry (canvas.stanford.edu/courses/233286) and was redirected to the Stanford login page, confirming the sign-in wall. Neither the course site nor a web search turned up a public YouTube or OCW recording, so there is nothing to embed.

Course and recording entries:

• Official course / lecture source
• Official Logistics page (Lecture Recordings section)

Checked: 2026-10-10.

Update log

• 2026-10-29: The series moved from one post per week to one post per lecture. This post now holds Lecture 18 (Mon Nov 30), previously part of the "Week 11: from waiting for instructions to acting first" guide; earlier entries below predate the split.
• 2026-10-23: Rebuilt the guide around source status, mechanisms, a worked scenario, and answer criteria; separated detailed assigned readings from lecture preparation.
• 2026-10-23: Added diagrams and comparison tables where useful for the existing explanation; this visual update does not reverify numerical results, schedules or research claims.
• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-09-12: Made the Week 10 Thanksgiving recess explicit in the series map and course-material checklist so the Week 9-to-11 jump no longer looks like a missing post.
• 2026-10-10: Rechecked video status. The official Logistics page says recordings require logging into the course Canvas, opening the entry confirmed the sign-in wall, and no public recording was found; the status is now sign-in required.

References

• On this site: Lecture 17: Coding & Software Agents, Lecture 19: Frontiers & Open Problems, Stanford CS329Z course overview
• Course: CS329Z schedule
• Sources: Shaikh et al., Creating General User Models from Computer Use, UIST 2025, Shaikh et al., Learning Next Action Predictors from Human-Computer Interaction, GUM project page and open-source package, Horvitz, Principles of Mixed-Initiative User Interfaces, CHI 1999