Skip to content

Long Context and SSMs: Different Ways to Retain History | CMU 10-423 Topic Guide

Sep 30, 20261 min
Home›ai
TL;DRLong-context evaluation asks whether a model can process, retrieve, and use distant information. An SSM specifies sequence processing through state updates. SSM efficiency does not establish reliable document retrieval, and a large attention window does not guarantee relevant evidence is used.
#cmu-10423#ai-course#cmu#long-context#attention#state-space-model#mamba#flashattention
Series: Reading CMU 10-423 Generative AI (43 / 47)
Distributed Training and Efficient Inference: Locate the Cost | CMU 10-423 Topic GuideOperational Risks and Alignment Theory: Tests and Assumptions | CMU 10-423 Topic Guide

Look up & save

Loading dictionary…

🌏 中文版

Video status: recordings require login or course authorization. Recording source

Long-context evaluation asks whether a model can process, retrieve, and use distant information. An SSM specifies sequence processing through state updates. SSM efficiency does not establish reliable document retrieval, and a large attention window does not guarantee relevant evidence is used.

Choose a lesson

The course map lists the core path. This page is a topic guide, not an additional lecture. Choose the question you need to answer and follow its lesson.

A worked comparison

Teaching SSM: h0=0, ht=0.5h(t−1)+xt, inputs [2,0,6,2], and output equal to state. The states are [2,1,6.5,5.25]. The first input contributes 2×0.5³=0.25 directly at step four. This decay follows our chosen coefficient, not every SSM. To evaluate document use, separately hold question and evidence fixed and move evidence between the beginning, middle, and end of a document.

Check your understanding

If the first input changes from two to four, how much does the fourth state increase?

Answer and criterion

It increases by 2×0.5³=0.25, from 5.25 to 5.5. Computing state influence still does not replace an empirical text-retrieval test.

How to use this map

Before reading the answer, identify the inputs, what changes, and the outputs. Then complete the linked lesson’s variation. If two mechanisms seem identical, trace where their conditions or data paths diverge. This page checks relationships and boundaries; it does not replace the derivations, implementations, or evaluations in the individual lessons.

The source edition is Spring 2026. Teaching examples are not official assessments or reconstructed classroom speech. Recordings remain subject to course authorization.

Course video sources

The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, the anonymous reader returned an unloaded folder and a sign-in prompt; it did not provide playable lecture content. This establishes an access limit, not the absence of recordings. This guide uses public written materials and provides no reconstructed classroom speech or original-audio transcript.

Update Log

  • 2026-10-11: Rebuilt the teaching scope, worked example, and self-check; retained the original URL and publication date.

  • 2026-10-10: Added explicit video status and checked recording sources and access notes.

  • 2026-10-10: Rechecked video status. The login wall is confirmed (anonymous Panopto folder lists no videos and prompts sign-in); the official pages link no public YouTube version, so the status is unchanged.

References

Related

Modern Transformers: What Position Rotation and KV Sharing Change | CMU 10-423 Lecture 41 minTesting minGPT Changes: Shapes, Masks, and Positions | CMU 10-423 Assignment 11 minCode Generation and Agents: A Reliable Test–Repair Loop | CMU 10-423 Lecture 231 min