RAG Prompt Engineering: How to Design System Prompts and Context

🌏 中文版

Your RAG search is working well, the context is accurate, but the LLM's answers are still underwhelming. The problem is usually in the prompt design — how you combine context and instructions to tell the LLM how to respond.

Basic Structure of a System Prompt

A RAG system prompt typically has three parts:

1. Role definition: Who you are, what you can do
2. Behavioral guidelines: How to use context, how to handle uncertainty
3. Output format: Structure and tone of responses

You are the NobodyClimb climbing knowledge assistant, familiar with crags, routes, and climbing techniques in Taiwan and around the world.

[Behavioral Guidelines]
- Only answer based on the provided knowledge base; do not add information beyond the knowledge base
- If the knowledge base has no relevant data, say "No information available on this topic" — do not guess
- Difficulty information must be precise; avoid vague expressions (use "5.11a" instead of "moderate difficulty")
- Safety-related advice should be conservative; when uncertain, recommend consulting an experienced coach

[Output Format]
- Respond in Traditional Chinese
- When recommending routes, list name, difficulty, and type
- Use bullet points where appropriate, but don't over-structure
- Keep answers concise; avoid being verbose

The role definition helps the LLM understand its position; behavioral guidelines prevent hallucinations and off-topic responses; the output format ensures consistency.

Context Formatting

How you format context affects how the LLM understands and cites information:

Poor formatting:

Context: Longdong North Wall 5.11a dense protection bolts clear fall zone Longdong South Wall 5.9 beginner route suitable for beginners Longdong Summit 5.12a technical route requires good footwork...

All documents are concatenated with no boundaries. The LLM struggles to distinguish content from different documents.

Good formatting:

[Knowledge Base]

Document 1:
Route name: Some Route (Longdong North Wall)
Difficulty: 5.11a, sport climbing
Description: Dense protection bolts, clear fall zone, crux move after the third bolt...
---

Document 2:
Route name: Another Route (Longdong South Wall)
Difficulty: 5.9, sport climbing
Description: Beginner route, suitable for newcomers, evenly spaced bolts...
---

Clear document boundaries (---), structured fields (name, difficulty, description) — these enable the LLM to accurately cite information from specific documents.

Document Relevance Hints

Adding relevance scores to context helps the LLM know which documents are more trustworthy:

[Knowledge Base] (sorted by relevance)

Document 1 (relevance: high):
...

Document 2 (relevance: medium):
...

Document 3 (relevance: low, for reference):
...

The LLM will tend to cite "relevance: high" documents more heavily, reducing the influence of low-quality context.

Instruction Placement

Lost in the Middle measures where relevant information (documents, key-value pairs) sits in the context, not where instructions sit. The "state it at the top and again at the bottom" habit below is closest to the paper's §4.2 query-aware contextualization, which the paper found minimally changes trends on multi-document QA and slightly hurts in most settings — so treat this as practitioner habit worth A/B testing, not a finding:

[System Prompt — Beginning]
You are a climbing assistant. Only answer based on the provided knowledge base.

[Knowledge Base]
...(large volume of documents)...

[User Message — End]
Please answer based on the knowledge base above: {query}
If the knowledge base has no relevant information, say "No information available on this topic."

The critical "do not hallucinate" instruction appears at both the beginning and end, so it won't be diluted by the long context in the middle.

How strong this positional bias is varies by model and by context length, and it varies a lot between vendors — you have to measure it on the model you actually use to know how much is left. But putting important instructions at the head and tail costs essentially nothing, so treat it as cheap insurance.

Chain-of-Thought (CoT)

For complex queries, add CoT instructions to have the LLM show its reasoning process:

Before answering, please:
1. Identify the core requirement of the question
2. Confirm which relevant materials exist in the knowledge base
3. Organize the answer based on relevant materials

Then provide the final answer.

CoT makes the LLM's reasoning more structured, especially when comparing multiple options or making recommendations.

Two caveats, though. First, the benefit of CoT correlates positively with model size, not negatively. The original paper's conclusion is that this reasoning ability "emerges naturally in sufficiently large language models"; small models asked to reason step by step often produce fluent but invalid chains that drag the answer off course. So do not assume "the model is small, therefore it needs CoT more." Second, for reasoning models that already think internally, the official guidance has a section headed Avoid chain-of-thought prompts, stating that telling the model to "think step by step" or "explain your reasoning" is unnecessary; the overview paragraph is more hedged, saying such techniques may not enhance performance (and can sometimes hinder it). The reasoning already happens inside the model, so the extra CoT instruction is usually redundant and sometimes harmful.

So CoT is worth putting into an A/B test, but it should not be a default-on best practice.

Common Issues and Fixes

Issue: The answer includes too much content beyond the context

Fix:

Strict constraint: Your answer must be 100% based on the provided knowledge base.
If the knowledge base lacks certain information, explicitly state that data is unavailable — do not supplement.

Issue: Inconsistent answer formatting

Fix: Provide a few-shot example:

Answer format example:
Q: What routes at Longdong are suitable for beginners?
A: Longdong has the following beginner-friendly routes:
1. **Some Route** - 5.9, sport climbing, dense protection bolts, suitable for first-time outdoor climbing
2. **Another Route** - 5.8, sport climbing, clear route, consistent difficulty

Please follow this format when answering.

Issue: Imprecise difficulty information (saying "moderate difficulty" instead of 5.10b)

Fix:

Difficulty must be expressed using precise climbing grade standards (e.g., 5.10b, V4).
Do not use vague terms like "moderate" or "fairly difficult."

Issue: Answers are too long

Fix:

Answer length limits:
- Simple questions (definitions, single facts): 1-2 sentences
- Recommendation questions: List 3-5 options with 1-2 sentence descriptions each
- Complex analysis: 200 words maximum
Avoid lengthy preambles and summaries.

Prompt Version Management

Prompts are code — they need version control:

const PROMPTS = {
  "v1.0": {
    system: "You are a climbing assistant...",
    contextTemplate: "Document {n}:\n{content}\n---",
  },
  "v1.1": {
    system: "You are a climbing assistant...(improved version)",
    contextTemplate: "Document {n} (relevance: {relevance}):\n{content}\n---",
  },
};

// Read the current version from ai_config
const currentVersion = config.prompt_version ?? "v1.1";
const prompt = PROMPTS[currentVersion];

Versioning makes A/B testing different prompts possible and makes rolling back to an older version easy.

Key Takeaways

Prompt Engineering is undervalued in RAG systems. No matter how good your search is, if the LLM doesn't know how to use the context — or is confused by poor prompt structure — answer quality will suffer.

The most effective prompt improvements:

1. Clear role definition (let the LLM know its boundaries)
2. Well-formatted context (document boundaries, structured fields)
3. Explicit "say you're uncertain when uncertain" instructions (prevent hallucinations)
4. Output format examples (few-shot is the most direct approach)

Get these four points right, and answer quality improves noticeably — no complex techniques required.

Changelog

• 2026-08-19: Fact-checked against primary sources and refreshed; perishable details handed back to official docs. Added to the "RAG Techniques Compendium" series.

References

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)
• Lost in the Middle: How Language Models Use Long Contexts (2023)
• Retrieval-Augmented Generation for Large Language Models: A Survey (2023)
• OpenAI - Reasoning best practices (reasoning models do not need CoT instructions)
• NobodyClimb System Architecture: Cloudflare Full-Stack Climbing Community Platform
• NobodyClimb AI Architecture: 20-Node RAG Pipeline