RAG Multi-Entity Queries: When the User Lists Five Routes and the System Only Sees the First

TL;DR The RAG system's extractRouteReference() used a for...return pattern that grabbed only the first match — so when a user provided five completed routes, only one was used. The fix evolves through three layers: rule-based multi-entity extraction, user profile aggregation, and embedding centroid.

#rag #ner #query-decomposition #recommendation-system #multi-hop-qa

Table of Contents

The Problem: The Cost of for...return
Solution Landscape
Multi-Hop QA: The Academic Connection
Chosen Approach: Layered Implementation
In Summary
Changelog
References

Series: The RAG Techniques Compendium (35 / 48)

"Recommend the next route" and "Recommend something similar" are not the same thing — Intent Disambiguation in RAG Recommendation Systems When Vector Search Matches by Name Instead of Grade: Attribute Conflation in RAG Systems

🌏 中文版

The user says: "I recently completed Baihu 5.11d, Lightning 5.12a, Looks Like I Can 5.11c, Bubblegum 5.11b, and Hsinchu Hakka 5.10d — recommend 3 routes I haven't climbed." The system sees Baihu and returns. The other four routes? Gone.

This is a real bug that surfaced in the NobodyClimb climbing recommendation system. This post documents the root cause, how academia and industry handle similar problems, and the layered solution we ultimately chose.

The Problem: The Cost of for...return

The logic in extractRouteReference() was simple: iterate over known route names, match against the query string, and return on the first hit. This works perfectly for single-entity queries like "recommend routes similar to Baihu." But when a user provides five routes at once, three kinds of information evaporate immediately:

Difficulty range loss — The five routes span 5.10d to 5.12a, meaning the user's ability covers three major grade levels. By grabbing only Baihu at 5.11d, the recommendation range gets pinned to 5.11a–5.12b (±3 steps), completely ignoring the fact that the user has already sent 5.12a.

Style preference loss — Multiple routes may be spread across different crags and types (sport / trad / boulder), and that distribution reflects climbing preferences. A single route cannot represent it.

Incomplete exclusion list — The user said "routes I haven't climbed," but excludeRouteId only excludes Baihu. The other four routes can still appear in the recommendation results.

In IR terminology, this is a complex information need. Metzler & Croft (2005)'s Markov Random Field retrieval model was built to break the convenient-but-wrong assumption that the terms in a query are independent of one another; a multi-entity query is the same problem scaled up — not just dependencies between terms, but several self-contained entities packed into one utterance. In the recommendation context this is the inverse of cold start: the user has proactively provided rich preference signals, and the system is consuming a fraction of them.

Solution Landscape

After surveying the literature, five approaches emerged (pun intended).

Multi-Entity Extraction

The most intuitive fix: change extractRouteReference() to extractRouteReferences() and return an array instead of a single result.

Rule-based approach: Replace for...return with for...push, adding a consumed-range mechanism to prevent overlapping matches (once "Looks Like I Can" is matched, "I Can" cannot match again). Aggregate the difficulty of all routes into a union range:

gradeFilter = {
  $gte: min(allGrades) - margin,
  $lte: max(allGrades) + margin
}

LLM-based approach: Have the LLM extract all routes at once via structured output:

{
  "routes": [
    {"name": "Baihu", "grade": "5.11d"},
    {"name": "Lightning", "grade": "5.12a"},
    {"name": "Looks Like I Can", "grade": "5.11c"},
    {"name": "Bubblegum", "grade": "5.11b"},
    {"name": "Hsinchu Hakka", "grade": "5.10d"}
  ],
  "intent": "recommend_next_challenge",
  "exclude_mentioned": true
}

On the academic side, Li et al. (2020)'s FLAT (Flat-Lattice Transformer) uses a lattice structure to consume characters and words together, and was SOTA on Chinese NER when published (later models have since moved past it, but the architectural idea still travels). Yan et al. (2021) unified flat, nested, and discontinuous NER into a single seq2seq generation problem, emitting all entities in one pass — which naturally supports multi-entity extraction. The industry equivalent is the joint intent-plus-entity model in NLU toolkits, e.g. Rasa's DIET (Dual Intent and Entity Transformer) and CRF entity extractor.

Query Decomposition

Decompose a complex query into multiple sub-queries, retrieve independently, then merge. For this case:

Profile Sub-query: Extract all five routes → build a user capability profile
Exclusion Sub-query: Exclude all five routes → build an exclusion list
Recommendation Sub-query: Use the profile as a basis → execute recommendation retrieval

There are several relevant frameworks. Self-Ask (Press et al., 2023) has the LLM ask itself follow-up questions, breaking a complex question into independently answerable sub-questions. IRCoT (Trivedi et al., 2023) goes further by interleaving Chain-of-Thought reasoning with retrieval — each reasoning step generates new retrieval needs. LangChain's Multi-Query Retriever and LlamaIndex's Sub-Question Query Engine follow the same paradigm. Note that as of LangChain 1.0 these legacy retrievers live in the langchain-classic package (SelfQueryRetriever's current API reference) — following an older tutorial's import path will just raise ImportError.

User Profile Aggregation

Rather than treating each route as a separate retrieval anchor, aggregate them into a user preference profile:

Dimension	Aggregation Method	Result for This Case
Ability ceiling	max grade	5.12a → recommend 5.12a–5.12c
Comfort zone	median grade	5.11c
Route type	sport/trad/boulder ratio	sport-leaning → weighted
Crag	crag distribution	multi-crag → no restriction
Exclusion	collect all route_ids	all 5 excluded

At the embedding level, compute a centroid:

query_vector = mean([embed(Baihu), embed(Lightning), embed(Looks Like I Can), embed(Bubblegum), embed(Hsinchu Hakka)])

This is the classic average pooling of item embeddings from recommendation systems. The YouTube Recommendations paper (Covington et al., 2016) uses the average of watch-history embeddings as the user representation — the exact same concept.

Plan-and-Execute RAG

Introduce a planning phase that analyzes the complete intent before executing step by step:

Query → Planner → [Step 1: Extract all routes]
                   [Step 2: Build user profile]
                   [Step 3: Determine search criteria]
                   [Step 4: Vector search with aggregated filter]
                   [Step 5: Re-rank and exclude]
               → Executor → Response

This connects to the existing LangGraph architecture's multi_tool path, but the key difference is that the planning phase in Plan-and-Execute is more structured (it doesn't just decide which tools to use — it also decides how to aggregate intermediate results), and the execution phase includes a feedback loop. Wang et al. (2023)'s Plan-and-Solve Prompting and Yao et al. (2023)'s ReAct point in this direction.

Collaborative Filtering

Look at "what other users who climbed the same routes also climbed":

-- Find users with similar taste
SELECT user_id, COUNT(*) as overlap
FROM ascents
WHERE route_id IN ('baihu_id', 'lightning_id', 'looks_like_i_can_id', 'bubblegum_id', 'hsinchu_hakka_id')
  AND user_id != current_user_id
GROUP BY user_id
ORDER BY overlap DESC
LIMIT 10;

-- Find recommendations from similar users
SELECT route_id, COUNT(*) as popularity
FROM ascents
WHERE user_id IN (top_10_similar_users)
  AND route_id NOT IN (excluded_routes)
GROUP BY route_id
ORDER BY popularity DESC
LIMIT 3;

The concept is elegant, but the constraints are practical: it requires sufficient user records, the computation is too heavy for Cloudflare Workers' CPU time limits, and it's better suited to offline precomputation rather than real-time queries.

Multi-Hop QA: The Academic Connection

Multi-entity queries are closely related to Multi-Hop QA. Key papers:

HotpotQA (Yang et al., 2018) — Multi-hop QA benchmark requiring reasoning across multiple documents
MDR (Xiong et al., 2021) — Iterative retrieval that adjusts each hop's query based on the previous hop's results
Baleen (Khattab et al., 2021) — Condensed retrieval that compresses intermediate results across hops for efficiency
DSP (Khattab et al., 2023) — Precursor to DSPy; demonstrate-search-predict pipeline

For Query Decomposition, Ma et al. (2023)'s Query Rewriting and Shao et al. (2023)'s ITER-RETGEN (iterative retrieval-generation) are worth studying. For multi-signal fusion in recommendation systems, SASRec (Kang & McAuley, 2018) uses self-attention over interaction sequences, and Li et al. (2023)'s GPT4Rec frames recommendation directly as a language task.

Chosen Approach: Layered Implementation

Given the constraints of the Cloudflare Workers runtime, D1 SQLite, and the existing LangGraph architecture, a single big-bang rewrite wasn't feasible. The approach breaks into three layers:

P0 (immediate): Multi-Entity Extraction

Smallest change, biggest impact. extractRouteReference() → extractRouteReferences(), returning an array. The caller aggregates filters across multiple routes, and excludeRouteId becomes excludeRouteIds. No additional LLM calls, no latency increase. Estimated to resolve 80% of multi-entity query problems.

P1 (short-term): User Profile Aggregation

Add buildUserProfile() to compute the ability ceiling, comfort zone, preferred type, and crag distribution from multiple routes. The profile is injected into the LLM prompt's system context, giving the model complete user background when generating responses.

P2 (medium-term): Embedding Centroid

If multiple routes have embeddings, compute the centroid vector as the query vector. Results are then re-ranked: exclude mentioned routes, apply difficulty-appropriateness weighting, and apply diversity weighting. Vectorize's query() already takes an arbitrary vector, so that part is not the obstacle; the real precondition is on the metadata side (see below).

Integration points with the existing architecture:

nlp.ts            → extractRouteReferences() returns array
tool-selection.ts → routeRefs: RouteReference[], aggregated filter
GraphState        → excludeRouteIds: string[]
filter-build.ts   → handles multiple crags ($in instead of $eq)
vector search     → post-filter excludes multiple routes ($nin instead of $ne)

Three hard Vectorize constraints are worth writing on a sticky note before you start — the official metadata filtering docs spell them out:

Metadata indexes must exist before you insert the vectors (wrangler vectorize create-metadata-index), and an index supports at most 10 of them. Deciding after the fact that you want to filter on a field means re-loading the whole vector set.
String metadata is only indexed on its first 64 bytes; anything past that cannot participate in a filter. A Chinese character is 3 bytes, so long route or crag names hit this quickly.
Range operators ($gt/$gte/$lt/$lte) may only be combined with each other, not with $in on the same key. So "grade as a union range" and "crag as an $in list" have to live on different fields — which our design already does, but do not expect to run a range and an allow-list on grade.

In Summary

The core trade-off is "maximize recommendation quality improvement with minimal additional computation, within edge runtime constraints." P0's rule-based multi-entity extraction is nearly zero-cost and resolves the majority of problems — the best return on investment. P1 adds profile aggregation so the LLM has fuller context, at the cost of a bit more string processing. P2's embedding centroid is where computation and API limits actually come into play, so feasibility validation is needed first.

Query Decomposition and Collaborative Filtering are deferred for now. The former overlaps with the existing LangGraph multi-tool path; the latter needs more user data. Once P0–P2 are live, the next step depends on where the recommendation quality bottleneck actually sits.

Changelog

2026-08-19: Fact-checked against primary sources and refreshed; perishable details handed back to official docs. Added to the "RAG Techniques Compendium" series.