Published on

Under the Hood of Retrieval-Augmented Generation

Authors
  • avatar
    Name
    Diego Carpintero
    Twitter

This article has been updated in August 2026

Introduction

Large Language Models (LLMs) operate under an inherent constraint as the factual knowledge encoded in their parameters is frozen at training time. To address this limitation and ground outputs in up-to-date, citable sources, Retrieval-Augmented Generation (RAG) extends the prompt with contextually relevant passages retrieved from an external knowledge base at inference time.

Introduced by Lewis et al. in 2020 [1], RAG was proposed as a solution for tasks where the required knowledge exceeds what can be learned by a model. By separating what the model knows from what it can look up, it demonstrated that retrieving the most relevant passages from an external memory reduces hallucinations, improves factual accuracy, and allows outputs to be traced back to their sources.

Typically, RAG pipelines comprise five stages:

  • document segmentation (or chunking): partitioning source documents into smaller chunks optimized for embedding and retrieval;
  • indexing: transforming chunks into dense embeddings and persisting them in a vector database;
  • retrieval: querying the vector index to identify candidate passages semantically aligned with the user prompt;
  • reranking: re-scoring retrieved candidates to prioritize the most relevant context; and
  • generation: injecting the highest-ranked passages into the prompt to ground and condition the model's response.

As model providers continue to expand supported context length into the millions of tokens [2] [3], it may be tempting to place an entire knowledge-base directly into the prompt instead of building a retrieval pipeline. In practice, however, long-context ingestion and RAG are complementary rather than competing strategies.

RAG generally outperforms pure long-context approaches for corpora exceeding 200K tokens (about 500 pages) [4], while remaining more cost-effective at any scale given that computational overhead grows non-linearly with context length. Beyond this threshold, longer prompts significantly impact Time-To-First-Token (TTFT) and introduce positional interference, wherein relevant information located toward the middle of a corpus receives diminished attention [5]. This results in retrieval performance degradation, even well before reaching nominal context limits [6] [7]. Production systems therefore tend to favor a hybrid architecture where RAG identifies the most relevant passages, over which long-context reasoning is applied.

Table of Contents

Document Segmentation (Chunking)

How a corpus is partitioned into chunks directly determines what the retrieval component can fetch, and consequently, what context the model receives at inference time. A suboptimal segmentation strategy will propagate silently while the system appears to function at every individual step, yet the model would generate responses conditioned on irrelevant or missed context, affecting both accuracy and completeness [8].

Chunking strategies fall into three primary categories based on how boundaries are determined: size, structure, or semantics. The appropriate choice depends on the nature of the data and expected queries. In practice, this involves some inherent trade-offs:

  • Retrieval Accuracy: smaller chunks produce more focused embeddings that match narrow queries more precisely, but risk fragmenting information across boundaries; larger chunks preserve cross-sentence dependencies but dilute the semantic meaning of embeddings, which reduces retrieval precision.
  • Inference-Cost vs. Indexing-Overhead: smaller chunks lower latency and token consumption at inference time; however, they increase the number of vectors to be generated, stored, and searched.
  • Scalability vs. Pipeline-Complexity: chunking enables corpora of arbitrary scale to be indexed and queried efficiently, but requires preprocessing, maintenance of a vector database, and a retrieval mechanism.

Fixed-Size Chunking with Overlap

Fixed-size chunking divides a document into contiguous segments of a predefined length, typically measured in characters, words, or tokens. When applying this strategy, the target chunk size must not exceed the maximum input length of the embedding model; a surplus will either trigger runtime errors or be silently truncated by the tokenizer.

To mitigate boundary degradation (such as splitting words or severing adjacent phrases), it is common to introduce a sliding overlap, where consecutive chunks share a fixed proportion of content (10–20%). This preserves semantic continuity and prevents context fragmentation. In production systems, fixed-size chunking with overlap is often the default choice as it is format-agnostic, straightforward to implement, and consistently delivers predictable retrieval performance.

def chunk_by_size(
    document: str,
    chunk_size: int = 512,
    chunk_overlap: int = 100,
) -> list[str]:
    """Split a document into fixed-size character chunks with sliding overlap."""

    if chunk_overlap >= chunk_size:
        raise ValueError(
            f"chunk_overlap ({chunk_overlap}) must be smaller than chunk_size ({chunk_size})"
        )
    if chunk_overlap < 0:
        raise ValueError(
            f"chunk_overlap ({chunk_overlap}) cannot be negative"
        )

    chunks = []
    i = 0

    while i < len(document):
        j = min(i + chunk_size, len(document))
        chunks.append(document[i:j])
        i = j - chunk_overlap if j < len(document) else len(document)
    
    return chunks

Structure-Aware Chunking

Rather than splitting on position, structure-aware chunking uses the document's own structure as boundaries such as markdown headings, paragraphs, HTML tags, or code block delimiters. Each chunk is thus more likely to encapsulate a self-contained semantic unit. This strategy produces the most coherent chunks when applied to well-formatted documents, which is however not always the case as documents might lack in practice structural markers.

Semantic Chunking

This approach aims at detecting semantic boundaries using the embedding space. The document is first segmented into sentences, which are then embedded independently. Consecutive embeddings are then compared and grouped if they are semantically similar. A split is introduced wherever similarity drops below a threshold.

Indexing

After chunking the document corpus, the next step is to persist the chunks in a form that supports efficient retrieval at inference time. In practice, each chunk is stored alongside its metadata and its corresponding embeddings. This enables the system to identify which chunks are semantically relevant to a given user query.

Embedding Transformation

A commonplace retrieval mechanism in RAG pipelines is semantic search. Unlike keyword-based search, which looks for exact matches between a user query and a source, semantic search operates over embeddings. This allows a query about "how does self-attention work?" to retrieve a chunk discussing "key, value optimization" even though the two texts share no tokens.

An embedding is a dense numerical vector to encode the semantic content of a piece of data (in our case text) through its position in a high-dimensional space. The model is trained such that semantically similar texts are mapped to nearby regions in the vector space, and dissimilar texts to distant regions.

Proximity is measured by cosine similarity (the cosine of the angle between two vectors), which is bounded between −1 and 1, where 1 indicates identical direction and 0 indicates orthogonality. Because embedding models are typically unit-normalized, cosine similarity is mathematically equivalent to dot-product similarity, and many implementations use the faster dot product operation. Retrieval systems often work with the complementary cosine distance, which converts similarity into a dissimilarity metric:

cosine distance = 1 - cosine similarity

It is worth noting that although these vectors encode meaning faithfully enough to support retrieval, individual dimensions are not directly interpretable. That is, a vector can be regarded as encoding many latent aspects of the text, but these aspects are not aligned with human-readable features, and no single dimension corresponds to a specific, interpretable property.

When implementing an embedding transformation, it is common to choose an embedding model; in our example, we rely on Voyage 4 [9]

import voyageai

def generate_embeddings(chunks):
    vo = voyageai.Client()
    # This will automatically use the environment variable VOYAGE_API_KEY.
    # Alternatively, use vo = voyageai.Client(api_key="<your secret key>")

    return vo.embed(chunks, model="voyage-4-large", input_type="document")

embeddings = generate_embeddings(chunks)

Vector Databases

A vector database is optimized for storing, indexing, and searching through said high-dimensional embeddings. Typically, they organize the information is into buckets, trees or graphs. Unlike traditional relational databases, which index scalar values for exact match queries, vector databases index high-dimensional vectors to support fast approximate nearest-neighbor (ANN) search over semantic similarity.

In a RAG pipeline, each chunk is embedded once during ingestion and then persisted together with its metadata and (optionally) the original text. Production systems typically store as metadata fields such as document ID, source URL, section title, timestamp, or access-control tags. These can later be used to constrain searches or enable filtering.

store = VectorIndex()

for embedding, chunk in zip(embeddings, chunks):
    store.add_vector(
        vector = embeddings, 
        metadata = {"content": chunk.content, "url": chunk.url}
    )

Retrieval

Embedding-based Retrieval

User queries are embedded using the same model as during indexing, producing a vector in the same high-dimensional space. The database then computes distances between the query vector and the indexed vectors, returning the top_k closest matches.

user_embedding = generate_embedding("What were the most relevant developments in AI last year?")

results = store.search(user_embedding, k=5)

for doc, distance in results:
    print(distance, "\n", doc["content"][0:200], "\n")

However, an exact nearest-neighbour search is not feasible at production scale as computing the distance between a query vector and every indexed embedding grows linearly with corpus size, which would be significantly expensive to implement. In practice, retrieval systems use Approximate Nearest Neighbour (ANN) algorithms, which trade a small reduction in recall for orders-of-magnitude improvements in query latency. Common ANN approaches include:

  • LSH (Locality-Sensitive Hashing): maps similar vectors to the same hash buckets with high probability, restricting the search to a smaller candidate set.
  • HNSW (Hierarchical Navigable Small World): constructs a multi-layer proximity graph over the embedding space where nodes represent vectors and edges connect similar vectors, enabling logarithmic-time traversal to approximate nearest neighbours.
  • Annoy (Approximate Nearest Neighbours Oh Yeah): partitions the embedding space into a forest of random projection trees, each providing an independent candidate set that is merged at query time.

Embedding-based retrieval excels when the query and relevant documents use different wording but share the same intent. However, it can underperform for queries that rely on exact tokens, such as IDs, error codes, or domain-specific terminology. For this reason, most production RAG systems combine multiple retrieval strategies, namely dense retrieval (embedding-based) and sparse retrieval (term-based), to balance semantic understanding with term-level precision.

Term-based Retrieval (Lexical Search)

The most straightforward way to find revelant information given a user query is with terms or key-words. However, a query may contain many terms, and terms may be present in multiple documents.

Intuitively, the more documents contain a term, the less informative that term is. Several methods build on this idea of term frequency:

  • TF-ID: combines term frequency (TF) and inverse document frequency (IDF) under the premise that a term's importance is inversely proportional to the number of documents it which it appears.
  • Elastic-Search: maintains an inverted index that maps terms to the documents that contain them, along with term frequencies and document counts.
  • Okapi BM25: scores documents based on term overlap, term frequency, and inverse document frequency. Unlike semantic search, it rewards exact matches and is particularly strong for queries containing identifiers, acronyms, or domain-specific terms.

Hybrid Search

Semantic search alone does not always return the best results. Sometimes exact term matches are needed (e.g., searching by ID) that semantic search might miss. A common solution is to combine both retrievals, semantic search with lexical search using a technique called hybrid search:

  • Semantic search finds conceptually related content using embeddings.
  • Lexical search (e.g. BM25) finds exact term matches using classic text search.
  • Hybrid search merges both signals to improve overall recall and precision.

Reciprocal Rank Fusion

The key challenge in hybrid search is that each retriever uses a different scoring scale (e.g., cosine similarity vs BM25 scores). Simply concatenating or averaging raw scores is therefore unreliable. Reciprocal Rank Fusion (RRF) addresses this by operating purely on ranks, which are comparable across methods. It merges and re-ranks results from multiple search methods by rewarding documents that appear near the top of multiple ranked lists.

Reranking

Retrieval is optimized for recall: quickly surfacing a broad set of candidate passages that might be relevant. Reranking is optimized for precision: carefully reordering those candidates so that only the most useful passages are included in the context passed to the generator.

This is typically implemented as a two-stage pipeline:

  • Retrieval: use one or more retrievers (semantic, BM25, or hybrid) to fetch a relatively large candidate set, e.g., the top 50–100 chunks.
  • Reranking: pass each (query, candidate) pair through a more powerful model such as a cross-encoder, to obtain fine-grained relevance scores; then retain only the top passages (e.g. 3-10) for generation.

Generation

At this point, the pipeline has already done the preparatory work: identifying relevant passages, scoring them for relevance, and selecting a small, high-quality subset. The LLM job is to synthesize this information into a coherent response while staying faithful to the provided sources.

How models arbitrate between parametric and contextual knowledge under conflict remains an active area of research. As of August 2026, context adoption appears to be shaped by a combination of factors: how coherent and plausible the content looks, the number of supporting documents, their position in the context window, and how confident the model is in its parametric knowledge. There is also no clear consensus on effective mitigations.

Model confidence is typically estimated using proxies such as token-level logits, consistency across paraphrased questions, and entity popularity in the training data.

Retrieval Optimization Techniques

Contextual Retrieval

If a document is split into multiple chunks, some chunks may lose their contextual relationship to the overall document, leading to retrieval failures. For example, a chunk reading "The company's revenue grew by 3% over the previous quarter" does not specify which company or period it refers to. Contextual Retrieval is a technique that enriches chunks with background information to improve matches for semantic queries.

In this regard, Anthropic [11] proposes generating a short context that explains the chunk and its relationship to the document, typically 50–100 tokens, and appending it to the chunk before embedding. In our example, the same revenue statement might become:

This chunk is from an SEC filing on ACME Corp’s performance in Q2 2023; the previous quarter’s revenue was $314 million. The company’s revenue grew by 3% over the previous quarter.

Microsoft [12] suggests instead to include structured metadata such as document title, summary, keywords, and hypothetical questions that the chunk can answer.

Note that while contextual retrieval improves accuracy, it comes with trade-offs as it increases both token-generation costs at indexing time and storage requirements in the vector database.

In the following implementation we opt for a hybrid approach that includes the contextual relationship and keywords to improve search retrieval:

def add_context(text_chunk, source_text):
    prompt = """
    You are tasked with generating a brief contextual snippet to improve
    search retrieval for a document chunk.
    
    <document>
    {text_document}
    </document>
    
    <chunk>
    {text_chunk}
    </chunk>
    
    Write a contextual snippet (50-100 tokens) that situates this chunk within the document. Include:
    - How this chunk relates to the document and section main topic
    - Key terms, entities, dates, or identifiers
    - Context that would help queries match this chunk
    Answer only with the succinct context and nothing else.
    """
    
    messages = []
    add_user_message(messages, prompt)
    result = chat(messages)
    
    return result["text"] + "\n" + text_chunk

References

  • [1] Lewis et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arxiv:2005.11401.
  • [2] OpenAI. 2025. Introducing GPT‑4.1 in the API openai:gpt4.1
  • [3] Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. meta:llama4.2025
  • [4] Anthropic. 2024. Introducing Contextual Retrieval. anthropic:2024
  • [5] Liu et al. 2023. Lost in the Middle: How Language Models Use Long Contexts. arxiv:2307.03172.
  • [6] Hong et al. 2025. Context Rot: How Increasing Input Tokens Impacts LLM Performance. hong:2025
  • [7] Modarressi et al. 2025. NoLiMa: Long-Context Evaluation Beyond Literal Matching icml:modarressi
  • [8] Barnett et al. 2024. Seven Failure Points When Engineering a Retrieval Augmented Generation System arxiv:2401.05856
  • [9] Voyage AI. 2026. The Voyage 4 model family voyage:2026
  • [10] Sanderson et al. 2012. The History of Information Retrieval Research. ieee:6182576
  • [11] Anthropic. 2024. Introducing Contextual Retrieval. anthropic:091924
  • [12] Microsoft. 2025. Design and develop a RAG solution microsoft:010925
@misc{carpintero-retrieval-augmented-generation,
  title = {Under the Hood of Retrieval-Augmented Generation},
  author = {Diego Carpintero},
  month = {ago},
  year = {2025},
  date = {2025-10-12},
  publisher = {https://tech.dcarpintero.com/},
  howpublished = {\url{https://tech.diegocarpintero.com/blog/retrieval-augmented-generation/}},
  keywords = {llms, rag, chunking, semantic-search, bm25, vector-storage, reranking, contextual-retrieval},
}
View on GitHub