Retrieval-Augmented Generation (RAG)

What It Is

Retrieve relevant information from external sources and use it to ground the model’s response.
RAG is not synonymous with long-term memory.
Information retrieval + context assembly + LLM generation

Why It Is Needed

Gaps in the Model’s Knowledge
- The knowledge stored in model parameters may be incomplete or outdated.
- Private or domain-specific knowledge may never have appeared in training data.
Context Window Limitations
- The available documents may exceed the model’s context window.
- A larger context window does not guarantee effective use of all the supplied information.
- Including too much irrelevant context raises cost and latency and can reduce answer quality.

How to Implement RAG

Overall Workflow
- Indexing Stage
  - Ingest source data → clean and split documents → create embeddings or indexes → store them
- Query Stage
  - Query → Retriever → Context Construction → Generator → Answer
Two key components
- Retriever
  - Classification
    - Lexical Retrieval
      - Implementations
        
        BM25
        
        Why It Is Needed
        
        TF-IDF limitations
        
        Word frequency linear growth: the more keywords appear, the fractions may be over-expanded
        
        Document length deviation: Long documents tend to have a higher number of keywords
        
        Implementation Principles
        
        Term-Frequency Saturation
        
        Assume the user search for GMP. / Document GMP appearances Simple TF / A / 3 / 3 / B / 30 / 30 / If you look only at TF, B scores 10 times A. / However, the fact that an article mentioned 30 GMPs does not mean that it is 10 times more relevant than the article mentioned three times. / Therefore, BM25 introduces word saturation (TF Satellite): / The increase in the frequency of words can still increase the rating, but the benefits of the increase are gradually diminishing.
        
        Document length homogenization (Document Length Norwegianization)
        
        Assumptions: / Documents Total Words GMP appearances / A / 100 / 3 / B / 1,000 / 3 / The same number of keywords appear in both documents. / But A is a short text that focuses on GMP, and B may be a technical note that contains many other topics. / If you read only the original word frequency, they score the same. / BM25 introduces document alignment (Document Length Normalization) to avoid the unreasonable scoring advantage of long documents simply because they contain more text. / Note that this does not mean that the short document is necessarily more relevant, but rather that the statistical deviation caused by the length of the document is adjusted.
        
        Formula
        
        BM25 Scoring: term frequency, rarity, and document length. / A simplified term contribution is Score(t,D) = IDF(t) × [f(t,D)(k₁+1)] / [f(t,D)+k₁(1−b+b|D|/avgdl)]. Here f(t,D) counts occurrences of term t in document D; |D| is the document length; avgdl is the corpus average; k₁ controls term-frequency saturation; and b controls document-length normalization. / The formula combines three ideas: inverse document frequency favors distinctive terms, term-frequency saturation gives diminishing returns for repeated occurrences, and length normalization prevents long documents from receiving an unfair advantage merely because they contain more words. / Example: search for GMP in a corpus of 100 documents, 10 of which contain that term; average document length is 100 words. Let k₁=1.2, b=0.75, and IDF be approximately 2.264 under the chosen common IDF convention. / Document A is 100 words long and contains GMP 3 times; B is 1,000 words and contains it 3 times; C is 100 words and contains it 30 times. Approximate BM25 scores: C 4.79 (rank 1), A 3.56 (rank 2), B 1.21 (rank 3). / A versus B illustrates length normalization: the same term count scores lower in a much longer document. A versus C illustrates saturation: 10 times as many occurrences raise the score by only about 35%, not 10 times. Frequency matters, but its effect should not grow without bound.
        
        What It Is
        
        Document relevance rating method based on words
        
        Improvement of traditional TF-IDF scoring methods
        
        TF-IDF
        
        What It Is
        
        The more frequently a word appears in a document, the less it appears in the entire document set, the more it represents the contents of the document.
        
        Measuring the relevance of documents to queries by combining word and counter-document frequencies
        
        Implementation Principles
        
        Question one: The importance of a document cannot be distinguished by a keyword match
        
        TF
        
        Assumed user search: / AI engineering / The search system may have found 100 articles containing both words. / However, the context window of the model is limited and it is not possible to insert all the articles. / Which articles should be given priority? / An intuitive idea: / The more frequently a keyword appears in an article, the more relevant the article may be. / This led to the introduction of the first indicator: / TF / is the number of times a word appears in a document. / For example: / Number of appearances of documents TF / Document A 10 10 / Document B 3 3 / Document C1 1 / Only consider TF, document A is most relevant.
        
        Question two: Not all keywords are equally important
        
        IDF
        
        The original examples used by the authors are: / Easy-to-follow recipes for Vietnamese food to cook at home / It means: a simple, learning family cooking recipe. / The author notes that in this query: / - Vietnamese, recipes: has a high amount of information. / - for, at: frequent and low information. / Why? / Because if a word appears in almost all documents, it is difficult to help us distinguish between documents. / This led to the introduction of the second indicator: / IDF (Inverse Document Frequency) / to measure how rare a word is in the entire document collection. / Word, occurrence, amount of information. / For almost all documents have low / Recipes Part of Document Containing / Very few Vietnamese documents contain high / Therefore: / TF measures the importance of a word to the current document, and IDF measures the ability of the word to distinguish between the entire document set. / Note that TF is only an instructive indicator of relevance and does not guarantee that the higher the frequency of words, the more relevant.
        
        Together.
        
        The author combines two indicators: / [ / \operatorname{TF\text{-}IDF}(t,D) / =\operatorname{TF}(t,D)\times\operatorname{IDF}(t) / ] / Of which: / 1. TF: Phrase frequency / [ / \operatorname{TF}(t,D)=f(t,D) / ] / Name(t)In Document(D)Number of occurrences. / 2. IDF: Counter-document frequency / The official formula in the book is: / [ / \operatorname{IDF}(t)=\log\left(\frac{N}{C(t)}\right) / ] / - (N): Total number of documents. / - (C(t)): Include words(t). / Final score of the document relative to Query / [ / \operatorname{Score}(D,Q)= / \sum_{t\in Q}f(t,D)\operatorname{IDF}(t) / ] / is the TF-IDF score for each word in Query to be added to the document. / Here is a book detail: the author refers to 10/5 = 2 directly as IDF in the text example, but the formal formula contains a logarithmic operation. In strict accordance with this formula, the IDF shall read(\log 2)Not two.
      - What It Is
        
        Retrieval based on key word matching and relevance rating
    - Retrieval based on embedded
      - Implementations
        
        Embedding
        
        What It Is
        
        Converts text to a vector indicating its semantics.
        
        Embedding Model
        
        Models to generate vectors
        
        Make semantic data more vector-like degrees
        
        Vector Search
        
        What It Is
        
        Finds data based on similarities between vectors.
        
        Similarity calculations
        
        Cosine Similarity (cosine symmetry)
        
        Search algorithm
        
        k-NN (nearest known)
        
        Similarity to all vector calculations
        
        Accurate but costed
        
        ANN (nearest neighbour)
        
        Use index to narrow the search
        
        Retrieving efficiency in return for partial accuracy
        
        Common algorithms
        
        LSH
        
        HNSW
        
        IVF
        
        PQ
        
        Annoy
      - What It Is
        
        is to convert the text to a vector, and then retrieve the semantic content through a vector similarity.
      - Why
        
        And vice versa.
        
        Word matching cannot directly represent semantic correlation
    - Mixed Search
      - What It Is
        
        Combining word search with embedded search
  - What It Is
    - Access to relevant information by user Query
  - Data sources
    - Document
    - Relationship Database
    - Search Engine
    - External API
    - Code repository
- Generator
  - What It Is
    - Model for using search results to generate answers
    - Usually it’s LLM itself.
- Indexer
  - What It Is
    - Processing and organization of external data and establishment of an index that allows efficient access to Retriever
  - Why It Is Needed
    - Avoid running all raw data through queries
    - Improved retrieval efficiency
    - Supporting different search methods
  - Implementation process
    - Data Ingestion
      - Obtain raw data from external data sources
    - Parsing
      - Convert raw data to indexable content
    - Chunging (parts)
      - Split long documents into data blocks suitable for retrieval
    - Index Building
      - Index based on words
        
        Phrase
        
        Create Backward Index
        
        Maintain word frequency and document statistics
      - Embedded Index
        
        Embeding Generates Document Vector
        
        Create vector index
        
        HNSW
        
        IVF
    - Storage
      - Save Index
      - Save raw data and metadata association
  - Index maintenance
    - Incremental Indexing
    - Reindexing
Detailed Execution Flow
- Indexing Stage
  - Access to external data
  - Data cleansing and segmenting
  - Indexing
    - Word Search
      - Create Backward Index
    - Embedded Search
      - Embeding Generates Document Vector
      - Create vector index
  - Storage index and raw data association
- Query Stage
  - Receive Query
  - Query rewrite (optional)
  - Retriever Search
    - Word Search
      - Keyword Matching and Relevance Rating
    - Embedded Search
      - Embeding Generate Query Vector
      - Victor Seach finds similar vectors
    - Mixed Search
      - Merge two-way search results
  - Reranking (optional)
  - Context Construction
    - Filter, organise search results
  - Generator
    - LLM generation of final answer

How to Optimize RAG

Purpose
- Supply the most relevant and useful evidence within the available context window
Indexing Stage
- Chunking
  - Problem Addressed
    - The document is too long and requires a reasonable cut.
  - Main trade-offs
    - Block Size
    - Semantic Integrity
    - Overlap Size
- Contextual Retrieval
  - Problem Addressed
    - The context of the original document is lost after the partition
  - Solutions
    - Adds background information or metadata from the original document to the segment
    - Indexing with enhanced content
Query Stage
- Reranking
  - Problem Addressed
    - The first search results were insufficiently sequenced
- Query Rewriting
  - Problem Addressed
    - User query incomplete or ambiguous

How to Evaluate RAG

Generation Quality
- Answer Correctness
  - Is the final answer correct?
- Factual Consistency
  - Is the answer consistent with the search?
Retrieval Quality
- Precision
  - How much of the search results are really relevant
- Recall
  - How many of all the results have been retrieved?
- Sort quality
  - Whether the relevant document is ahead