Building and evaluating RAG systems the right way

Retrieval-Augmented Generation, introduced by Lewis et al. in 2020, has become a core strategy for organizations deploying large language models (LLMs) in production. Its value lies in letting models use trusted internal data, improving factual grounding, and reducing risk without the overhead of training specialized models. When implemented well, RAG extends general-purpose LLMs into domain-specific enterprise tasks while controlling cost and complexity.

Many organizations discover that initial performance does not hold once the system meets real-world usage. Answers fluctuate, relevance decays, and confidence stays high even as accuracy drops. These issues get attributed to hallucinations, but hallucinations are usually a symptom rather than the cause. Most failures arise from weaknesses in retrieval, ranking, context construction, or inconsistent operational processes, and the absence of a structured evaluation framework makes it hard to detect where the pipeline is drifting or why performance is unstable.

A RAG system is not a monolithic component. It is a sequence of interconnected decisions that must remain coherent for the system to behave reliably. Each stage introduces dependencies, potential failure modes, and opportunities for drift. Building reliable RAG systems requires understanding these components not only in isolation but also in how they interact over time.

The sections below provide a detailed overview of the RAG architecture, identify where failures typically occur, and outline a rigorous evaluation methodology that organizations can apply to maintain system quality in production.

Core components of a RAG system

While RAG can be conceptually described as “retrieve relevant data, then generate an answer,” the operational reality is more complex. A well-structured RAG pipeline typically includes the following components:

• Text-splitter
• Embedder
• Vector database
• Retriever
• Reranker
• Generator (LLM)

Click to expand

Each plays a specific role, each introduces risks, and each requires clear performance expectations and evaluation metrics.

Text splitter

The pipeline begins with splitting documents into smaller segments. This step determines the structural granularity through which the model will perceive the corpus. Although the mechanics seem straightforward, this is often one of the earliest points where quality begins to degrade.

Smaller chunks reduce embedding cost, improve granularity, and reduce the risk of exceeding context limits. However, they may fragment cohesive ideas, weakening semantic coherence and reducing retrieval effectiveness. Larger chunks preserve context but dilute semantic clarity, increase computational overhead, and often reduce retriever precision because meaningful signals become harder to match. Overlapping windows help maintain continuity but increase disk usage, index size, and latency.

Poor chunking is one of the most common sources of weak retrieval, and its impact is usually misattributed to the model. A well-designed text-splitting strategy should reflect the document’s structure, domain, and intended retrieval patterns.

Embedder

Embedding is the process of converting text into vector representations that encode latent semantic relationships. These embeddings form the backbone of similarity search and significantly influence retrieval quality. Three factors are especially important:

1. Dimensionality: Higher-dimensional models capture richer nuance but increase storage requirements and query latency. Organizations must balance representational power with operational cost.
2. Model consistency: When embedding models change, even slightly, the geometric structure of the vector space changes along with them. If the corpus is not re-embedded and re-indexed, retrieval precision drops due to misaligned vectors. This is often called embedding drift.
3. Query - document parity: Queries and documents must be embedded using the same model and configuration. Any deviation creates inconsistency, leading to unpredictable retrieval outcomes.

Embedding strategy requires clear versioning, careful change management, and coordination with downstream components. The same machinery extends to images, charts, and tables in multimodal RAG, where a shared embedder places text and visuals in one vector space.

Vector database

Vector databases store embeddings and support efficient similarity search across large corpora. Their behavior is shaped by both the volume and distribution of the data they index.

Key considerations include:

1. Index performance: Indexes degrade as the corpus size and distribution change. Periodic rebuilds are required to maintain search accuracy and latency.
2. Latency and throughput: Query speed depends on dimensionality, index structure, hardware, and concurrency patterns.
3. Mixed-version embedding risks: Storing embeddings generated from different model versions within a single index reduces similarity scoring fidelity and significantly weakens recall.

A well-designed vector storage system must be monitored continuously and refreshed as data, models, and usage patterns evolve.

Retrieval

The retriever selects the top-k chunks most relevant to the user’s query. This stage heavily influences system performance because it governs what information the LLM can base its answer on. Selecting k requires careful balance:

• Too few items and essential information may be omitted.
• Too many items and the LLM receives noisy context, weakening grounding and increasing hallucination likelihood.

This reflects the classical precision-recall trade-off. Retrieval quality is also impacted by chunking strategy, embedding consistency, index performance, and corpus coverage.

Production RAG systems often employ hybrid retrieval, combining dense vector search with sparse keyword search or metadata filtering, to strengthen recall across varied query types.

Reranking

Once retrieved, candidate chunks are re-scored by a reranker model that evaluates query-document interactions more deeply. Cross-encoder rerankers often outperform dense retrieval by incorporating richer contextual signals. For accurate scoring, the reranker must receive both the chunks and the query. It then assigns relevance scores, and the highest-ranked items form the final context for the LLM. Reranking mitigates many issues that arise from imperfect retrieval. When well configured, it is one of the highest-impact components in an RAG system.

Generation

The generation step uses the retrieved context and the query to produce a response. Model selection affects tone, reasoning depth, and structural capabilities, but the quality of grounding depends heavily on upstream components. Even with accurate retrieval, models may generate unsupported content if the prompts fail to emphasize adherence to context or if the retrieved evidence is insufficient. Prompt design, refusal behavior, and context formatting all play meaningful roles in the reliability of generation.

Connecting the pipeline

The RAG pipeline consists of two parallel flows:

1. Document upload flow: When a user provides a document to a chatbot or any other kind of RAG system, the first step is reading the textual content on the page. This text is then chunked using text-splitter. Text chunks are then forwarded to the embedder, where they are embedded and turned into vectors. In the end, those vectors are written in a vector database.
2. Query flow: When a user asks a question related to the uploaded document, this query is embedded using the embedder in the same way as text from the uploaded document. An embedded representation of the query is sent to the vector database, where a similarity-based search is performed. Only the k most relevant context vectors are returned to the LLM along with the query, so the LLM can generate an answer based on the retrieved context, though the quality of the answer still depends on retrieval accuracy.

RAG capabilities can look simple, but their impact on the final decision goes well beyond providing information, because they let general-purpose LLMs answer domain-specific questions. This means you do not have to retrain or fine-tune a general-knowledge model on domain-specific data every time the need arises, though highly specialized domains may still call for light fine-tuning for reasoning or structured-output tasks. A high-quality RAG system carries that knowledge instead.

Evaluating RAG systems

A structured diagnostic framework

To operate a RAG system reliably, teams must measure where and how the pipeline performs. Each component has associated metrics that reveal specific failure modes. Without these metrics, system degradation becomes difficult to diagnose. Evaluation metrics fall into two categories, retrieval-based and generation-based.

Retrieval-based metrics

Contextual relevancy

This metric measures how much context the RAG system provides that is relevant to the query and the LLM response. Since the main purpose of the RAG system is to provide LLM with the most relevant content from input data, it’s obvious why this metric takes such an important place in the evaluation pipeline. The more relevant the context, the better the quality of the answer.

How is this metric actually calculated? Natural language carries many interpretations of the same idea, so computing relevance or similarity between two objects is hard, and pure vector similarity is insufficient for the semantic relevance RAG evaluation needs. A common alternative is LLM-as-a-judge, where an external LLM determines the relevance of the provided context to the user’s query and the model’s output.

The process has three steps:

1. Extracting statements from provided contextual data pieces.
2. Calculating how relevant those statements are to the input query and the LLM’s response.
3. Computing contextual relevancy score as the ratio between the number of relevant context pieces and the total number of provided context pieces.

Providing the most relevant data to the LLM is the first step to successful answer generation.

Contextual precision

While contextual relevance assesses how relevant the provided context is to the user’s query and the generated answer, contextual precision checks the order in which the contextual pieces are retrieved. Since the most important task of the RAG system is to rank contextual pieces in the database and provide LLM with the k most relevant ones, it is important to check that those data points are ranked correctly.

The core idea of contextual precision is to check if retrieved chunks are the ones labeled in the ground truth, and then if those vectors are ranked correctly. The final score indicates whether vectors are ranked in the correct order, from most to least relevant.

Contextual recall

Contextual recall calculates how many of all the relevant vectors the system actually retrieves. The number of retrieved vectors is always less than or equal to the predefined value of k, and not all of them will be relevant to the question asked. Determining how many of the retrieved vectors are labeled as relevant in the ground truth tells you whether your ranker and retriever are performing as desired.

Unlike contextual relevance, contextual precision and contextual recall are metrics that typically require ground-truth information. But in real-world scenarios, ground truth is not always available, as it requires investing additional working hours to provide reliable context. Since large documents can contain many vectors, this job can take hours or even days.

So, to escape this kind of grind, you can calculate contextual precision and recall using LLM-as-a-judge methods, which compare retrieved vectors with the user’s input to assess their relevance. And there are a few tools that can help, such as DeepEval, Ragas, RagaAI, etc.

Generation-based evaluation metrics

Evaluating a RAG system requires more than assessing retrieval quality. The final outputs must also be examined to determine whether the large language model is using the retrieved context effectively and responding in a manner that is factual, safe, and consistent with organizational requirements. The five metrics that matter most for assessing generation behavior are faithfulness, correctness, hallucination, toxicity, and bias. Each highlights a distinct dimension of model reliability.

Faithfulness

Faithfulness measures the degree to which the generated answer remains grounded in the retrieved context. A response is considered faithful when all claims, explanations, and references can be traced directly to the supporting evidence supplied by the RAG pipeline.

In practice, teams often use an LLM-as-a-judge mechanism to evaluate faithfulness. The judge model reviews the answer alongside the retrieved context and determines whether the generated statements are substantiated. High faithfulness signals that the model is adhering to the information provided, whereas low faithfulness typically indicates issues upstream, most commonly retrieval noise, incomplete context, or ineffective ranking. Faithfulness is foundational, and without strong grounding, no amount of model sophistication can compensate for weak evidence.

Correctness

Correctness measures whether the answer is factually accurate, independent of the provided context. The distinction matters because a model can produce a response that is entirely faithful to the retrieved documents while those documents are themselves outdated, incomplete, or incorrect. Correctness must therefore be evaluated separately from faithfulness.

High correctness indicates both that the model respects the context and that the underlying data is accurate and its reasoning aligns with the truth. When correctness is low but faithfulness is high, the responsibility shifts to data quality and knowledge governance rather than model behavior. Correctness and faithfulness complement one another, and both are essential for dependable RAG performance.

Hallucination

Hallucination captures the extent to which a model introduces unsupported or fabricated content. It occurs when the model generates statements that are not grounded in the retrieved evidence, or when it extrapolates beyond what the context can justify. Hallucination reflects the statistical nature of language models, which predict the most likely continuation given a set of tokens even when information is incomplete, so this metric evaluates how often and how severely the model departs from verifiable facts.

Modern approaches increasingly rely on verifier models that review generated statements against the retrieved context or external authoritative sources. These verification layers identify fabricated claims and reduce the operational risk of deploying LLMs in high-stakes environments. Hallucination remains one of the hardest issues in production systems, and reducing it meaningfully lifts user trust and overall system reliability.

Toxicity

Toxicity measures whether the generated content includes harmful, offensive, or unsafe language. Because many foundation models are trained on large public corpora, they may inadvertently reproduce undesirable patterns from the underlying data. Monitoring toxicity is essential for any organization deploying LLMs at scale, particularly in customer-facing applications. High toxicity scores can create reputational risk, compromise user safety, and raise regulatory concerns. While toxicity does not measure accuracy, it defines whether the system behaves responsibly and aligns with organizational values. Toxicity evaluation should therefore be integrated into routine testing, particularly after model upgrades or prompt changes.

Bias

Bias measures the degree to which a model produces uneven or systematically skewed responses across demographic groups, topics, or scenarios. It reflects how well a system generalizes across diverse populations and whether it exhibits preferential behavior based on sensitive attributes.

For example, a model trained on unrepresentative medical images may perform well on one population but poorly on others. Such behavior undermines fairness, reliability, and operational performance. Biased models often fail when exposed to real-world diversity, even if they perform strongly in controlled tests. Bias evaluation is not only a technical requirement but a governance expectation. Addressing it proactively strengthens trust, reduces downstream risk, and supports responsible AI deployment.

Mapping metrics to failure modes

Below is a list of metrics mapped to their failure modes. This should serve as a quick checklist for engineering leaders to identify possible reasons the metrics are undesirable.

1. Low contextual relevancy = poor chunking, embedding drift, indexing issues
2. Low precision = the retriever is including irrelevant items
3. Low recall = retriever or index missing relevant items
4. Good retrieval metrics but weak answers = ranking or generation issue
5. High hallucination with good retrieval = prompt misconfiguration or model limitations
6. Low faithfulness = reranker or retriever passing noisy context
7. Low correctness with high faithfulness = incorrect or outdated source data

Reliable online tools for RAG evaluation

There are many tools that can help you in this process by providing functions for the described methods and automating evaluation pipelines. Some of the most popular ones are:

DeepEval is an open-source evaluation framework designed to simplify building and iterating on LLM applications. It lets you unit test LLM output while offering more than 30 evaluation metrics out of the box. DeepEval supports both end-to-end and component-level evaluation across RAG systems, agents, chatbots, and virtually any LLM use case. It also includes advanced synthetic dataset generation, highly customisable metrics, and tools for red teaming and safety scanning to identify vulnerabilities in your product.

RAGAS is designed for a systematic evaluation process of LLM applications. The library supports a workflow in which you can run evaluations, observe outcomes, and iterate based on consistent evidence. Ragas also allows you to define custom metrics or use the built-in ones, and includes convenient tools for dataset management and result tracking. With integrations for frameworks like LangChain and LlamaIndex, it can be easily incorporated into existing development pipelines.

TruLens is an open-source evaluation and monitoring library for large language model (LLM) applications and AI agents. It is especially useful for building and evaluating RAG systems, agents, chatbots, or summarizers. It offers a variety of built-in feedback functions, such as context relevance, groundedness, answer relevance, coherence, toxicity, sentiment, and bias, to systematically score and trace every run. It can also be integrated with telemetry standards for production-ready tracing, and it supports popular stacks like LangChain and LlamaIndex.

The evaluation process has never been easier to implement and use in all types of LLM applications. Evaluating a wide range of metrics leads to more stable, production-ready pipelines, and many tools today offer a wide variety of metrics and tracking frameworks that help you achieve the best possible results and user satisfaction.

The path to dependable RAG systems

Building a reliable RAG system is ultimately a question of engineering coherence. Every stage, splitting, embedding, retrieval, ranking, and generation, contributes to the fidelity of the final answer. When even one component drifts, the entire pipeline reflects that drift. And without structured evaluation in place, this degradation remains invisible until it affects end users.

Organizations that approach RAG with discipline, treating retrieval as a measurable subsystem, embeddings as versioned assets, and evaluation as an ongoing operational requirement, gain far more than incremental improvements in accuracy. They establish predictability, build resilience, and create the confidence needed to scale AI across high-stakes workflows. These systems are sensitive by design, which means they must be governed with rigour, monitored continuously, and improved systematically. When implemented with this mindset, RAG becomes a durable system that strengthens decision-making and creates measurable strategic advantage.

Algorithmic designs and deploys RAG systems and knowledge bases built for production accuracy, governance, and scale. Start a conversation if your retrieval pipeline needs structured evaluation or operational hardening.

If you’d like to follow our research and case studies, connect with us on LinkedIn, Instagram, Facebook, X or simply write to us at info@algorithmic.co