Retrieval-Augmented Generation (RAG) Explained: Why Most Systems Fail and the 20 Components That Fix Them

RAG in one paragraph

Retrieval-Augmented Generation (RAG) is a method that brings together information retrieval and large language models to make responses more accurate and relevant. Rather than depending only on what the model already knows, RAG first finds useful documents or data from outside sources and uses them to help generate answers. This approach helps reduce mistakes and makes sure answers are based on current or specific information, while including only what is needed for the response. RAG is also often displayed with a simple diagram:

Unlock the potential of large language models

But this description and this simplest schema are misleading, especially when using any RAG-as-a-service tool without customisation or deeper diving.

RAG fails in the wild

Everyone seems to be betting big on RAG, with the market expected to reach $3.33 billion, but "big investment" and "actually works" aren't the same thing. Here we’ve listed some situations and quotes born from underestimating RAG, or simply not knowing its aspects.

• "Engineers built a $100K RAG system that performs worse than grep with an intern."
• "RAG woes: They were out of Ice Cube, so Claude got Vanilla Ice-perfect metaphor for substitution hallucinations."
• "RAG: Expensive keyword search with a side of hallucinations-because why fix retrieval when you can blame the LLM?"
• "Embeddings die in RAG purgatory: "$45.2M" chunk matches "$45,200,000" vibe? Nope, total mismatch."
• "Engineers survive RAG hell: Chunk, embed, rerank, pray. Repeat."
• "RAG is just fancy search... that costs 10x more and lies confidently."
• "After being told the microwave was purchased in 2025 and today's date was 2026, a local RAG model concluded: "The microwave is less than one day old."
• "Our retrieval is highly relevant to something. We’re just not sure to what."
• "Production RAG: Retrieves 50 docs on "Paris vacation," generates tips for Paris Hilton’s vacation wardrobe. Context window buried the travel bits."
• "RAG’s reasoning: "Here's your evidence... from a fanfic forum!"
• "RAG at a party: Someone whispers "ignore prior instructions" in the background docs. Suddenly, it's generating cat memes for your tax query."
• "Our RAG system is self-evaluating. It gave itself 10/10."
• "Free assistant hack: just ask a car dealer chatbot unrelated questions - it refuses to hang up."
• "Latency is just the number of unnecessary tool calls multiplied by confidence."
• "Context is king. That’s why we keep duplicating it across every MCP call."

Let's proceed directly to our topic: what is missing, why the RAG is more complicated and how to decrease the failure probability.

Key RAG components (beyond retriever and generator)

Here is the right time to ask when we need such components or what forces us to develop and include them in our RAG-based solution. Let’s go through them, describing the challenges that trigger component development and recommendations; each section also contains illustrative examples of what is being described.

1. Information decoding and normalisation

Every RAG pipeline starts with data ingestion, but most enterprise data is far from clean. This component is key for turning messy, noisy, and often ambiguously structured source data into a format that works for chunking, retrieval, and generation. In real-world cases, like scanned documents, semi-structured reports, or multilingual collections, raw data often has errors, structural issues, and hidden meanings that need to be fixed to avoid poor retrieval. Careful design of decoding and normalisation pipelines is needed to keep the RAG system accurate and reliable.

Main challenges:

• Automatic text recognition (ATR), especially for handwritten text (HTR), often leads to recognition errors and uncertainty.
• Complicated docs or tables with implicit relationships and non-linear layouts.
• Ambiguous language, including nested conditions, cross-references, and context-dependent meanings.

Here're samples of what enterprise data can actually look like once you start digging in:

Build your RAG-based solution

Unlock the full potential of your data

Recommended actions and submodules/nodes:

• Noise removal to structure the information (main signal w/o existing text enrichments), eliminate ATR artefacts and irrelevant symbols, etc.
• Focus on colontitles and other doc layers to better capture the structure and semantic anchors.
• Preserving semantic structure (sections, headers, footnotes, table relations) rather than flattening content into plain text.
• Unit normalisation (currencies, measurement units, time zones, accounting standards) to ensure consistency across sources.
• Attaching structured metadata (source, timestamp, author, version, jurisdiction) for further traceability and filtering.
• Verification of outdated materials and duplicates to maintain data relevance and integrity.
• De-identification and sensitive data masking or replacement when the vector database operates in an untrusted domain (can be developed as a separate module; we’ll focus on this later).

2. Adaptive chunking

The chunking component breaks input data into useful parts for retrieval, while trying to keep its structure and meaning intact. Unlike static methods, adaptive chunking considers differences between documents, hidden connections, and changes in content density. This makes it essential for keeping context relevant and retrieval accurate in RAG-based systems.

Main challenges:

• Under- or over-segmentation (chunk size imbalance, no matching to results/expectations).
• Loss of structural semantics and context, and topic drift within chunks (above picture with recursive chunking).
• Cross-references between sections.
• Query-agnostic nature of chunking (independent from user queries).
• Table and numeric data segmentation issues.

You can use ChunkViz, an open-source web tool, to check simple recursive chunking.

Recommended actions and submodules/nodes:

• Validate chunk boundaries and sizes to avoid fragmentation or excessive aggregation.
• Apply adaptive chunking (~recursive + structural + semantic) as the core mechanism:
  • Recursive splitting to balance chunk size dynamically.
  • Structural-aware segmentation aligned with document elements.
  • Semantic grouping to maintain topical coherence within chunks.

3. Knowledge graph augmentation

This component (or type - GraphRAG) enhances retrieval by introducing structured, relational representations of knowledge, enabling the system to move beyond isolated text chunks toward interconnected entities and multi-hop reasoning. In fields with high interdependency between concepts, such augmentation becomes essential for improving disambiguation, contextual completeness, and explainability of results. However, effective integration of knowledge graphs into RAG pipelines requires careful handling of data consistency, scalability, and representation strategies.

Main challenges:

• Merging of heterogeneous data sources, which often introduces inconsistencies in schemas, naming conventions, and levels of granularity across entities and relations.
• Entity (nodes) matching and disambiguation, where the same entity may appear in multiple forms, or identical labels may refer to different real-world objects depending on context.
• Graph explosion, caused by uncontrolled growth in the number of nodes and edges, leading to increased noise, redundancy, and computational complexity.

Recommended actions and submodules/nodes:

• Detailed ontology and schema design to standardise entity types, relationships, and constraints, ensuring consistency and interoperability across the graph.
• Dependencies extraction module to identify and formalise relationships between entities from unstructured and semi-structured data.
• Representation layer for how knowledge is stored (e.g., triples, embeddings, hybrid formats) while balancing search and comparison efficiency.
• Path-based reasoning (or path ranking approaches) to enable multi-hop approaches and identification of relevant relational paths between entities.
• As an option, connect your KAG task to some underlying GNNs to mimic graph structure and support context-aware, relation-aware predictive analytics (besides the retrieval).

4. Query correction

Evaluating and correcting user queries is important for matching user input with the system’s knowledge base. This helps make retrieval more reliable, even when queries are unclear, incomplete, or contain mistakes. In real situations, people often enter queries that do not follow the expected format because of typos, specialised terms, or missing context. These issues can lower retrieval accuracy; adding query correction tools can help improve precision in RAG-based systems.

Main challenges:

• Misspellings and typos, which distort query terms and reduce matching accuracy with indexed data (retrieval part).
• Use of acronyms and abbreviations, often domain-specific, that may not directly correspond to stored representations.
• Incomplete queries, lacking sufficient context or key entities required for accurate retrieval.
• Polysemy, where a single term may have multiple meanings depending on context.
• Misinterpretation of dates and numeric values, including formats, ranges, and units.

Recommended actions and submodules/nodes:

• Misspelling module to detect and correct typographical errors using lexical similarity or language models.
• Dictionary mapping to expand or normalise acronyms, abbreviations, and domain-specific terminology into canonical forms.
• Clarification (and memory augmentation) module/node to resolve ambiguity, enrich incomplete queries, and incorporate conversational context when available.

5. Query decomposition and enrichment

Further query processing helps turn complex, unclear, or multi-intent queries into structured, meaningful forms that retrieval systems can handle well. In practice, user queries often include hidden logic, lack context, or mix several goals. If these are not broken down and clarified, the results may be less accurate or even misleading. This step ensures queries are clear and match how information is organised.

Main challenges:

• Logical sequence and splitting are important because if decomposition is done incorrectly, it can break dependencies between sub-queries or disrupt the intended flow of reasoning.
• Semantic drift often happens after a query is broken down or transformed, causing the meaning to change from the original.
• Incomplete or unclear intent occurs when a query does not provide enough detail or includes assumptions that are not directly stated.

Recommended actions and submodules/nodes:

• Evaluate the granularity of the query and its intent to decide how much to break it down and to identify separate sub-tasks.
• Use HyDE and step-back prompting techniques to create intermediate representations or broader contextual queries, which help make retrieval more reliable.
• A clarification and memory augmentation module or node can help resolve ambiguities, add missing context, and use previous interactions when available, similar to the query correction task.

6. Embeddings and metric learning

Estimating similarity is important for filling in context, as it affects how well a system finds and ranks relevant results. In practical settings, standard embedding methods often miss complex meanings (firstly due to pooling/sentence embedding), precise numbers, or logical patterns in queries and documents. To better match user intent with stored information, it helps to design embedding strategies and similarity metrics carefully.

Main challenges:

• Semantic averaging (e.g., pooling, one embedding per large chunks), which compresses diverse information into a single vector and may obscure important nuances or minority signals.
• Poor numeric encoding happens when embeddings do not accurately represent numbers, ranges, or comparisons.
• Embeddings often struggle to tell the difference between positive and negative statements or to capture logical relationships.
• Cosine similarity has limits and may not capture complex semantic relationships or match domain-specific similarity needs.
• Multi-objective similarity is challenging because relevance often depends on several factors, like meaning, structure, or numbers, which are hard to capture in one embedding space.

Recommended actions and submodules/nodes:

• Use custom similarity scoring to include extra signals, such as structure, numbers, or metadata, instead of relying only on cosine similarity.
• Retrain embeddings so that models better fit the language, data, and tasks specific to your domain.
• Try multi-representation methods, like poly-encoders or multi-head encoders, or use multi-aspect embeddings to capture different meanings and make retrieval more reliable.

7. Search optimisation

Optimisation speeds up retrieval in tasks with large, complex search spaces that may need several steps of reasoning. As RAG systems grow, simple search methods become costly and less effective, especially with complex queries. So, optimisation techniques are needed to balance speed, relevance, and coverage.

Main challenges:

• High-dimensional and multi-hop real-time search, where queries must navigate complex embedding spaces and potentially require retrieval steps across related chunks (or whole vector DWH).
• A large search space can increase computational cost, latency, and risk of overload or weakly related candidates.

Recommended actions and submodules/nodes:

• Using ontology-based search methods like KAG or GraphRAG can help narrow down and guide retrieval, which reduces the effective search space.
• Unsupervised methods (grouping) to cluster chunks, enabling more efficient search through pre-grouped (~centroids-based) or hierarchical structures.

8. Advanced retrieval

Now that we have discussed separate components that support efficient retrieval, let's look at how exact retrieval works. Advanced or custom retrieval goes further than basic chunking and cosine similarity by using more sophisticated methods. These methods can handle complex, distributed, and varied information. In real-world situations, important knowledge is often spread across different sources, shown in different formats, or linked in subtle ways. Because of this, advanced retrieval systems are needed to provide thorough, accurate, and context-aware access to information.

Main challenges:

• Distributed information, where relevant data is scattered across multiple documents, sources, or storage systems, making single-step retrieval insufficient.
• Relational dependencies and cross-references, requiring the system to account for links between entities, sections, or documents.
• Numeric similarity vs. cosine similarity (e.g., "5% growth" vs. "50% growth"), where semantic similarity does not align with quantitative correctness.
• Heterogeneous data and modalities, including text, tables, structured data, and potentially other formats, which require unified retrieval strategies.
• Boilerplate dominance, where frequently repeated or generic content may overshadow more relevant but less frequent information during retrieval.

Recommended actions and submodules/nodes:

• Active retrieval (agentic retrieval) to dynamically guide the retrieval process based on intermediate results and planned steps.
• Multi-hop retrieval focused on iteratively gathering information across multiple chunks/sources, enabling reconstruction of complex answers.
• Contextual retrieval to incorporate surrounding context, metadata, and query intent for more precise matching.
• Hybrid retrieval combining multiple strategies (e.g., keyword-based (~best match), semantic, structured) to improve robustness and coverage.
• Federated retrieval to query and aggregate results from multiple distributed data sources or systems while maintaining consistency.

9. Reranking

The reranking component acts as a second step after the initial retrieval. Here, candidate results are reviewed and sorted based on how relevant they are to the task or query, not just on their embedding similarity. In real RAG systems, nearest-neighbour retrieval often brings up results that are related in meaning but not suitable for the context, so they cannot be used to answer queries. Reranking helps improve retrieval accuracy, cuts down on irrelevant results, and lowers the chance of producing responses that are incorrect or not well supported.

Main challenges:

• Nearest neighbours precision limitations, where top retrieved chunks may be semantically close in vector space but not sufficiently relevant to the actual query intent.
• Embedding similarity ≠ task relevance, since high cosine similarity does not necessarily correspond to factual usefulness, reasoning value, or importance.
• Multi-aspect queries, where relevance depends on simultaneously satisfying several conditions, constraints, or semantic dimensions.
• Boilerplate dominance, where repetitive/generic content receives disproportionately high retrieval scores due to frequent semantic overlap.
• Chunking noise, introduced by imperfect segmentation that produces incomplete, overlapping, or weakly coherent chunks.
• Hallucination risks, when low-quality or weakly related retrieved content becomes grounding material for generation.

Harness autonomous AI agents

Recommended actions and submodules/nodes:

• Use classical pairwise scoring to compare queries and documents directly, which helps estimate relevance beyond just vector similarity.
• Apply Learning to Rank (LTR) models that combine signals like semantic similarity, metadata, structural relevance, and user feedback.
• Use LLM-based reranking to take advantage of deep contextual understanding and task-aware reasoning when prioritising candidates.
• Apply contrastive reranking with strong negatives or adversarial examples to better distinguish between truly relevant and deceptively similar candidates.
• Use hybrid reranking methods that combine lexical, semantic, structural, and model-based signals to make ranking more robust and accurate.

10. Memory augmentation

The next section builds on RAG systems by helping them keep track of context across many interactions. This makes it easier to handle complex and repeated tasks in a more logical way. In real-world use, what the user wants often changes slowly over a series of prompts, so important information can end up spread across different steps or sources. Without memory features, these systems might lose track of context, make mistakes, or repeat the same searches. That’s why memory-aware tools are important for keeping things consistent, making the experience more personal, and improving how well information is found.

Main challenges:

• Incomplete queries, where user input lacks sufficient detail and depends on previously established conversational or task context.
• Progressive specification and clarification (chain of prompts), where intent evolves incrementally across multiple interactions rather than being fully expressed in a single query.
• Multi-step and multi-source tasks with multi-hop reasoning, requiring persistent tracking of intermediate states, retrieved evidence, and reasoning dependencies.
• Temporal context drift, where older context becomes outdated, less relevant, or conflicting with newly introduced information.
• Contradictory results, arising from inconsistencies between retrieved sources, prior memory states, or evolving user requirements.
• Computing and retrieval budget optimisation, where repeated retrieval of previously processed information increases latency and resource consumption (+ further budgets on cached input processing by LLM).

Recommended actions and submodules/nodes:

• Refine the query context step by step to improve and adjust the active context as the user's intent becomes clearer during the interaction.
• Recognise and extract user preferences, at least to the agent state, so the system can remember user-specific constraints, priorities, and behaviour patterns for future reasoning.
• Use smart memory buffer management to keep, prioritise, summarise, and remove information as needed, making sure the most relevant details stay available for current tasks.

11. Context compression

In large-scale RAG systems, retrieval pipelines may produce numerous overlapping or partially relevant pieces of evidence, especially in multi-task or multi-hop scenarios. Since downstream language models operate under limited context windows and computational constraints, efficient compression mechanisms are required to maximise information density, minimise irrelevant or redundant content and optimise computing budgets.

Main challenges:

• Candidates and evidence aggregation, especially in multitasking or multi-hop retrieval scenarios, where large amounts of partially overlapping context must be consolidated into a coherent representation.
• Limited context window, restricting the amount of information that can be passed to the generation model at inference time.
• Redundant and boilerplate corpora, where repetitive templates/reports, duplicated fragments, or generic content dominate the retrieved efficient context.
• Computing budget constraints, requiring minimisation of token usage, memory consumption, and inference latency.

Recommended actions and submodules/nodes:

• Use basic compression methods to turn large or wordy data structures into smaller, more efficient forms.
• Apply custom summarisation tools to keep important information for the task while making the context smaller.
• Use semantic deduplication to find and remove repeated or similar pieces of information in the collected evidence.
• Use query-focused compression to highlight information that is most relevant to the current question, task, or goal.
• Noise removal and instruction optimisation/compression to eliminate irrelevant text, formatting artefacts, or excessive prompt overhead.
• Use config compression to improve how you store and transfer extra settings, reasoning steps, skills, or metadata when needed.

12. De-identification module

It is important to handle sensitive information securely and in line with regulations when using RAG-based systems, especially if external APIs, third-party LLMs, or distributed retrieval setups are involved. In many enterprise or regulated settings, the data retrieved may include personal, confidential, or business-sensitive details that should not be shared with external services or unauthorised users. To reduce privacy risks while keeping the data useful for retrieval and reasoning, de-identification methods are needed.

Main challenges:

• Third-party LLM and API usage constraints, where sensitive information cannot be transmitted outside trusted environments or organisational boundaries.
• Regulatory compliance requirements, including obligations related to privacy, confidentiality, and data protection standards.
• Removing/hiding the sensitive information may change the context semantics and bring inaccuracies in the RAG responses.
• Internal access controls can be a challenge because different users, agents, or services may need different levels of access to sensitive content.
• Other forms of data leakage risks include unintended exposure through retrieved context, embeddings, logs, or intermediate reasoning states.

Protect your data with anonymisation

Recommended actions and submodules/nodes:

• Use entity recognition modules to find personal, confidential, or sensitive information in text, tables, and metadata.
• Apply smart masking to hide or change sensitive values, while keeping the text readable and consistent.
• Use data synthesis or replacement techniques to substitute sensitive information with realistic but non-identifiable placeholders or synthetic equivalents (probably the best one).
• Use smart encryption (including format-preserving approaches) to secure sensitive data while maintaining compatibility with downstream processing pipelines.
• Apply embeddings perturbation or obfuscation methods to reduce the risk of sensitive information reconstruction from vector representations (mostly local/open-source models).

13. Response synthesis

In the final stage of the RAG pipeline, a node takes the retrieved evidence and turns it into clear, well-supported, and relevant responses. In real systems, the information that is retrieved can be fragmented, sometimes even contradictory, or spread across different sources. This means the system needs to be able to combine and reason over this information. Also, responses should follow a consistent style, know when to end the conversation, and include an appropriate closing message.

Main challenges:

• Multi-step tasks and multitask queries, where the final response must combine information obtained across several reasoning stages or heterogeneous retrieval results.
• Knowledge staleness (outdated information), where retrieved candidates may no longer reflect the latest state of the domain or external environment.
• Bias propagation, where biases present in source documents, retrieval strategies, or model priors influence the generated response.
• Conversation closure ambiguity, where the system may fail to determine when sufficient information has been provided, how to gracefully conclude the interaction, or when additional clarification, verification, or follow-up actions are still required.

Recommended actions and submodules/nodes:

• Apply multi-hop reasoning methods to help bring together and resolve information that comes from different sources or reasoning paths.
• Use iterative generation approaches (e.g. fusion-in-decoder and related strategies) to progressively refine response drafts using multiple retrieval results and intermediate synthesis stages.
• Ensemble generation to combine outputs from multiple specialised models or reasoning nodes for improved robustness and coverage.
• Exit message mechanisms (additional conditional edges) to explicitly indicate uncertainty, insufficient evidence, retrieval limitations, or potential ambiguities when reliable synthesis cannot be guaranteed.

14. Caching

Nowadays, FinOps is essential for optimising cloud spending (in this context: more about tokens usage), ensuring computing resources are used efficiently (local LMs), and keeping technology costs aligned with business goals and budget constraints. Thus, caching is important for reducing redundant computation, decreasing response latency, and lowering expenditures in production RAG pipelines. Since a substantial share of user requests are repeated or highly similar (especially in FAQ-like systems, where 60%+ of requests are repeatable), repeated execution of the retrieval, reranking, and generation stages can be inefficient and unnecessary. Therefore, caching mechanisms should be introduced to reuse previously computed results whenever possible.

Main challenges:

• Computational budget constraints, since repeated retrieval and generation steps increase token usage (LLM-level caching also brings costs), memory consumption, and infrastructure cost.
• Latency requirements, where every additional retrieval step or node adds delay and degrades user experience.
• Repeated or highly similar queries, because a significant share of requests may be identical or near-duplicate, making full recomputation inefficient.

Protect your enterprise data

Optimise your cloud costs

Recommended actions and submodules/nodes:

• Use Q→R caching to store and reuse final query-to-response mappings for repeated or near-repeated requests.
• Set up FAQ or dictionary-based caching as a basic layer for queries or lookups that are frequent, stable, and easy to predict.
• Apply embedding-based caching for Q→retrieval/context→R flows, especially with local LLM scenarios, where cached embeddings or retrieval contexts can be matched against similar inputs.
• Tools/agents nodes caching to reuse intermediate outputs from tool calls, agent steps, or orchestration nodes when the same subtask appears again.

15. Multi-node system and orchestration

The orchestrator, along with other nodes and conditional edges, coordinates how retrieval, reasoning, external tools, databases, and execution environments work together, enabling the RAG system to operate as a dynamic multi-stage pipeline rather than a single retrieval-generation process. In complex enterprise and agentic AI scenarios, answering a query may require some calculations or code execution, and iterative reasoning across several interconnected modules. Therefore, orchestration mechanisms and a multimode pipeline are required to manage workflow control, task decomposition, execution dependencies, and adaptive decision-making.

Main challenges:

• Additional logic, machine learning operations (model-based agents), or external calculations that cannot be handled through retrieval alone and require integration with dedicated processing nodes.
• Database queries (besides semantic retrieval) and interactions with structured storage systems, where retrieval must be combined with transactional or analytical operations.
• Code execution requirements, including runtime computations, data transformations, or programmatic validations during reasoning workflows.
• Dynamic decision-making, where the system must adapt graph execution based on intermediate outputs, agent states or contextual conditions.
• Task decomposition and monitoring, especially in complex multi-step workflows requiring coordination, dependency tracking, and execution supervision.

Recommended actions and submodules/nodes:

• Use graph-based workflow orchestration, such as state machines or DAGs, to show how tasks flow, depend on each other, branch, and coordinate between processing nodes.
• Add collaboration and tool selection nodes that can choose the right tools, agents, or retrieval methods for each subtask as needed.
• Include ReAct-based reasoning to combine step-by-step thinking with action, so the system can adapt as it moves between retrieving data, using tools, and generating results.

16. Input-output verification

This is an additional wrapper around user input and system output that evaluates whether user input matches allowed usage and is compliant (+ jailbreaking prevention), whether generated responses are logically consistent, and whether they are reliable for downstream use. In practical RAG systems, even a well-formulated query and high-quality retrieval do not fully eliminate the risk of hallucinations, reasoning errors, and further biased conclusions introduced during generation. These risks become especially critical in high-stakes domains such as healthcare, finance, law, or industrial systems, where not efficient queries and corresponding incorrect outputs may lead to significant operational or regulatory consequences. Therefore, dedicated verification mechanisms are required before final delivery.

Main challenges:

• Hallucinations, where the generation model produces unsupported facts, fabricated details, or reasoning not grounded in retrieved evidence.
• Evidence-response mismatch, when generated conclusions only partially correspond to retrieved context or introduce unsupported extrapolations.
• Influence of LLM general/prior knowledge, where the model supplements retrieved information with pretrained assumptions that may conflict with current evidence.
• Conclusion and reasoning errors, including invalid inferences, omitted constraints, or logically inconsistent synthesis.
• LLM-as-a-judge weakness, since verification performed by the same or similar models may inherit the same biases and reasoning limitations as the original generator.
• High-stakes industry requirements, where outputs must satisfy strict standards of explainability, traceability, reliability, and compliance.
• Jailbreaks and prompt injection attempts, where malicious or manipulative input may override system instructions, bypass safeguards, or influence retrieval and reasoning behaviour.
• Improper usage and unsafe input patterns, including unsupported requests that may lead to usage abuse (and further FinOps/PR risks), unstable system behaviour or unreliable outputs.

Adopt machine learning solutions

Recommended actions and submodules/nodes:

• Use NLI (Natural Language Inference) modules to check for entailment, contradiction, and neutrality between inputs, retrieved evidence, and generated responses.
• Validate evidence (origin checks) and omission rate to confirm the provenance, reliability, freshness, and authenticity of retrieved information sources.
• Use entity comparison modules to make sure names, values, dates, quantities, and relationships are consistent between the evidence and the generated output.
• Apply DL-based verificators to give independent validation signals for both inputs and outputs, using dedicated verification or classification models.
• Include LLM-as-a-judge mechanisms as extra verification layers, but keep in mind they might block execution or have the same weaknesses as the main generation model in the RAG pipeline.

17. Guardrails

Besides the i/o verification, it is necessary to have a dedicated security layer, which typically acts as a safety, governance, and behavioural control layer for the RAG pipeline, ensuring that both retrieval and generation processes remain aligned with predefined operational, ethical, and domain-specific constraints. In practical deployments, RAG systems interact with untrusted user input, heterogeneous data sources, and dynamically changing contexts, which creates risks related to manipulation, unsafe outputs, and uncontrolled behaviour adaptation. Therefore, guardrail mechanisms are essential for maintaining reliability, policy compliance, and controlled reasoning boundaries throughout the system's usage.

Even the most advanced AI can be manipulated with the right wisper

Main challenges:

• Prompt injection (direct/indirect) and adversarial manipulation, where malicious instructions attempt to override system behaviour, manipulate retrieval logic, or bypass safety constraints.
• Non-compliant content generation, including outputs that violate regulatory, ethical, organisational, or domain-specific requirements.
• Knowledge scope and domain boundary violations, where the model generates responses outside approved domains (or retrieval results), trusted knowledge areas, or authorised contexts.
• Behaviour drift, where model responses gradually diverge from intended behaviour patterns due to accumulated context, memory effects, or adversarial interactions.

Recommended actions and submodules/nodes:

• Use knowledge limitation mechanisms, including instruction-based gating (verification node separation from input nodes) and selective LLM unlearning, to constrain reasoning and generation within approved boundaries.
• Add style-following nodes and instruction control layers to make sure communication patterns, response formats, and behavioural policies are followed as planned.
• Include security shields and use red teaming approaches, which are often handled by LLM providers, to find vulnerabilities, adversarial patterns, and unsafe behaviours early.
• Apply constitutional AI methods to guide generation with set principles, policy limits, and self-critique tools.
• Use heuristic filters to spot and block suspicious prompts, unsafe outputs, or unusual interaction patterns before sending the final response.

18. Evaluation/validation

An evaluation mechanism is essential for systematically assessing the quality, robustness, and reliability of the entire RAG pipeline, including retrieval, reasoning, orchestration, guardrailing and generation stages. In practical RAG systems, failures are often distributed across multiple interconnected components, making root-cause analysis difficult and increasing the risk of unnoticed degradation after updates or configuration changes. Furthermore, there are typical business requests about agent improvement (especially self- or semi-supervised), and they aren’t possible (any evaluation-improvement loops) without performance measurement and gaps/drifts detection.

Main challenges:

• Errors isolation difficulties, since failures are often connected to specific RAG components (retrieval, chunking, reranking, synthesis, orchestration, etc.) and may propagate across the pipeline.
• Misalignment and hallucinations, where generated responses deviate from retrieved evidence, intended behaviour, or task requirements.
• Sensitivity to prompt design, chunking strategies, and embedding configurations, which may significantly influence retrieval quality and downstream reasoning behaviour.
• Regression risks arising from changes to LLM versions, prompts, configurations, retrieval logic, or indexed data sources.
• Stochastic nature of generation models, where repeated executions may produce different outputs under identical conditions, complicating reproducibility and benchmarking (measurement).

Build AI solution you can trust

Recommended actions and submodules/nodes:

• Start with deterministic metrics to give stable and repeatable evaluation results that do not depend on generation randomness whenever possible.
• Use adversarial and stress testing to check how robust the system is against prompt injections, edge cases, noisy data, and unusual operating conditions.
• Apply testset-based and trace-based evaluations to check RAG system behaviour with curated benchmarks and real workflow execution traces.
• Include NPS (~top2box) and HIL (Human-in-the-Loop) evaluation mechanisms to measure user satisfaction (if the users don’t/can’t use the system, all other metrics don’t matter), expert review, and manual validation into quality assessment processes.
• Use component-based evaluation strategies to independently assess retrieval, reranking, reasoning, guardrailing, orchestration, and generation modules for more precise failure localisation.
• Use LLM-as-a-judge approaches as supplementary evaluation mechanisms, while accounting for potential biases, inconsistency, and shared weaknesses with the evaluated models. Please be sure that the metrics are well defined (not just “evaluate smth from 0 to 100”).

19. Feedback and improvements

This component is partially connected with the previous section and is responsible for continuous adaptation and optimisation of the RAG pipeline based on user interactions, observed failures, evolving datasets, and changing usage patterns. Unlike traditional ML systems with relatively stable training and inference boundaries, RAG architectures combine different components and layers, making iterative improvement significantly more complex. In practical environments, both the underlying knowledge base and user behaviour evolve over time (different types of drift), requiring adaptive mechanisms capable of continuously refining prompts, retrieval strategies, evaluation criteria, and model behaviour.

Main challenges:

• Differences from traditional ML solutions, since RAG systems combine multiple connected components whose behaviour depends on retrieval quality, prompting/query, orchestration, and external knowledge sources rather than only model weights.
• Data shifts affecting both indexed knowledge and user query distributions, potentially reducing retrieval relevance and response quality over time.
• Delayed dissatisfaction or misunderstanding, where users may not immediately recognise low-quality outputs, hallucinations, or incomplete reasoning.
• Nondeterministic feedback in textual form, where user feedback is often ambiguous, inconsistent, indirect, or difficult to automatically formalise into optimisation signals.

Recommended actions and submodules/nodes:

• Testset gathering and versioning mechanisms to continuously collect representative evaluation scenarios and track behavioural changes across pipeline versions.
• Use automated prompt and configuration revision approaches (e.g., BootstrapFewShot, MIPROv2, GEPA) to iteratively optimise prompts, orchestration logic, and retrieval settings based on observed performance. Check out the research paper from ELEKS experts with a structured comparison of automated instruction revision and other task adaptation strategies: Automated Instruction Revision (AIR).
• Include smart HIL (Human-in-the-Loop) mechanisms to add expert review, selective validation, and guided correction workflows into continuous improvement processes.
• Apply ReAct-based iterative reasoning and correction loops to enable adaptive refinement of retrieval and agent behaviour during task execution.
• Underlying LLM tuning approaches (although often computationally inefficient and can significantly decrease general LLM performance), including RLHF, RLAIF, and policy optimisation methods such as PPO or GRPO, to align generation behaviour with desired objectives and operational constraints.

20. Monitoring and communication

The final part is typical of any agentic solution, including RAG-based agents, and is responsible for gathering the necessary information and enabling reliable communication between components, agents, and external services. In complex multi-agent or multimodal systems, the lack of visibility into intermediate states, execution paths, and node interactions can make debugging, scaling, and coordination extremely difficult. Therefore, monitoring and communication mechanisms are required to ensure traceability, orchestration quality, and safe interaction across distributed parts of the system.

Main challenges:

• Bottleneck detection and understanding, where performance degradation may occur in RAG components, but the exact source is difficult to identify without proper observability.
• Lack of traceability, which limits the ability to reconstruct execution paths, analyse intermediate agent states, and explain why a specific output was produced.
• Poor coordination or redundant nodes, where overlapping functionality, inefficient routing, or unstructured communication leads to unnecessary latency and resource consumption.
• Multi-agentic solutions versus multimodal agents, where different execution paradigms require different monitoring and communication strategies, but both may introduce additional complexity in coordination and debugging.

Recommended actions and submodules/nodes:

• All components should have traces and APIs to make intermediate states, inputs, outputs, and decisions observable across the full pipeline.
• Use smart and efficient conditional edges to route execution only when needed, reducing redundant calls and improving operational efficiency.
• Apply state-based orchestration to keep a clear view of workflow progress, agent status, and task dependencies while running processes.
• Use event-driven and end-to-end tracing to capture transitions, interactions, and failures across the full lifecycle of a request.
• Set up secured MCPs/A2As/ANPs/ACPs (FastAPI-based) or similar frameworks to support controlled communication, secure tool access, and structured interaction between distributed nodes and agents.

Practical recommendations for RAG implementation

Presented RAG components demonstrate that modern retrieval-augmented generation systems are no longer simple retrieval-and-generation pipelines, but rather complex distributed architectures combining information retrieval, reasoning, orchestration, validation, security, and continuous optimisation mechanisms. Each component introduces its own set of challenges related to semantics, scalability, reliability, explainability, and operational safety, while also strongly interacting with other parts of the system. As a result, effective RAG implementation requires not only advanced language models, but also carefully designed supporting infrastructure, evaluation strategies, and control layers.

We know that it is hard to implement all possible components, especially when users don’t need them; thus, it is important for AI experts to understand and make decisions. Below are simplified recommendations which may be useful:

1. Do not fully rely on simple built-ins (recursive chunking, one-embedding cosine-based retrieval, answering with candidate summarisation, etc.).
2. Create and validate your RAG components based on actual needs (not simply because there is no such functionality in “framework”).
3. Do not use LLM-based approaches everywhere (for data transformation, evaluation, information retrieval, predictions, etc.).
4. Understand computational budgets and acceptable latency.
5. Understand the risks and mitigation strategies.
6. Create and use a test set.
7. Evaluate all available RAG steps.