LLMs Demystified:
Tokens, Context Windows & RAG.
System Architecture: Safe LLM Pipeline
01.LLMs Predict Tokens, Not Sentences
At their core, Large Language Models (LLMs) like Anthropic Claude, OpenAI GPT-4o, and DeepSeek are next-token prediction engines. They do not store a static database of facts; rather, they encode statistical relationships across trillions of text fragments called tokens (roughly 4 characters in English).
Because generation is probabilistic, the model samples words based on weights adjusted during pre-training. If asked a question without external grounding, it produces fluent, highly persuasive answers that might be factually fabricated — the phenomenon known as hallucination.
02.The Context Window: Working Memory
The context window is the maximum amount of input (prompts, conversation history, uploaded documents) that the model can process in a single inference call.
03.RAG: The Anti-Hallucination Weapon
Instead of fine-tuning models (which is slow, costly, and leads to stale knowledge), modern enterprise architecture uses Retrieval-Augmented Generation (RAG):
- Retrieve: When a user asks a question, an external vector database or search index fetches the top 3-5 verified source paragraphs.
- Augment: The prompt is injected with those paragraphs under a strict prompt rule: "Answer ONLY using the provided text."
- Generate: The model synthesizes the answer from the provided evidence.
This pattern powers real-world applications like my Singapore exam companion, Professor Turtle, ensuring generated practice questions adhere faithfully to the Ministry of Education syllabus.
The 90-Second Rule of Thumb:
Never trust an LLM to remember facts. Trust it to reason, transform, summarize, and extract over verified information you supply in its context window.