In our previous blog, Retrieval-Augmented Generation (RAG): An Executive Guide to AI Accuracy, we covered various aspects from what is RAG to why traditional models suffer from hallucinations to why hallucinations stop mattering.

In this blog, we cover how Lizard Global integrates it, using our latest project ZENO to explain it in detail. We sat down with in-house expert, Chathuri Senanayake, Head of Engineering to get the full breakdown.

Let’s get started.

In plain English, how does RAG actually stop the AI from making things up?

Standard LLMs generate a response by predicting the most statistically likely next word, based on the patterns learned during training. If it doesn't know, it will still produce a confident-sounding answer, which we call hallucinations.

RAG does not eliminate, but reduces hallucinations by grounding the model’s answer in real information by retrieving, augmenting and generating.

Retrieve: Before an answer is produced, the system searches internal documents available to it and retrieves content that is within a predefined similarity score. (We can adjust this score experimentally until we get the desired outcome.)

Augment: The retrieved content is inserted into a prompt along with a strict set of instructions and rules and sent to an LLM. We are now equipping the model with subject specific information. This strengthens the credibility of the response. Depending on the project we can explicitly set a rule to instruct the LLM to respond with “I don’t know” when the answer is not available in the context provided.

Generation: The LLM now produces a human readable response using the retrieved context and the instructions as its primary source, rather than solely relying on the training data.

While RAG does not completely remove hallucination, it reduces it by basing the response on verifiable information rather than a model’s best guess. It is important to note that the system is only as good as the information it retrieves and the AI model chosen: weak or irrelevant retrieval will still produce a bad, or hallucinated answer even with perfect prompting downstream.

How do we prepare our internal enterprise data so the RAG system can actually find the right information?

This step is the most important in a RAG system. A great AI model with a great instruction set will still fail if it's searching over badly prepared data. We consider the following steps when preparing the data through robust data and architecture design.

1. Clean and standardised data

Removing noise (redundant, repetitive) and converting them into machine readable (markdown where possible) is an important step of preparing the data.

2. Chunk the data

We don't hand documents to the AI as a whole. We break them into smaller, digestible pieces called chunks. How we chunk data depends on the type of project and type of information that the project consists of.

Common chunking methods include fixed-size chunking, semantic chunking and hierarchical chunking. We also often enrich chunks with metadata (dates, title, category) to optimize retrieval. Sited Source

3. Search strategy.

The traditional search strategy for the RAG model is semantic search. What this means is that a chunk of information is converted into a vector which is essentially a list of numbers that represent the meaning of the text. Words or concepts with similar meaning end up with numbers that sit close together (for example, "invoice," "bill," and "receipt" would cluster near one another). This is then stored in a vector database.

When a user question comes into the system, it's converted into a vector the same way, compared against everything in the database, and the closest matches are returned based on a similarity score. The similarity score can be tuned and will differ based on the project and its content.

While semantic search is powerful, it also makes sense to first filter data based on metadata. This hybrid search combines conceptual understanding and exact match precision.

To wrap up that point, Chathuri emphasizes that even the strongest AI models fall short without properly prepared data. Preparing internal enterprise knowledge effectively comes down to standardizing raw content into clean, machine-readable formats, breaking text into structured chunks enriched with metadata, and combining semantic vector similarity with exact metadata filtering for a high-precision hybrid search.

What happens when someone asks a question that isn't answered in our company documents?

This is where the guardrails such as threshold, instructions and confidence checks come into place. It is worth noting that there can be two failure modes:

1) The semantic search (Retrieval phase) finds nothing relevant based on the similarity threshold that was set. We often stop here before generation even happens and returns with a fallback response. This also prevents an unnecessary call to the LLM. We can still hand this over to a human in charge to notify that a question went unanswered, and whether an information update is required.

3) The semantic search finds something that is typically related but does not contain facts about what is asked. This is where a weaker AI model and/or an instruction set could still facilitate the AI to generate a weaker response. Therefore choosing the right model and strengthening instruction sets and rules is essential. Additionally a confidence score can be another signal to identify the strength of the response.

It’s important to monitor and log such low confidence responses and no responses such that the RAG model can be further strengthened.

In short, when handling unscripted questions, guardrails like similarity thresholds and confidence scores prevent bad outputs across two failure modes: returning a fallback before calling the LLM if retrieval fails, and using strict prompt instructions to avoid weak responses when retrieved text lacks exact facts. Chathuri noted that logging both fallback and low-confidence instances is key to flagging knowledge gaps for human review and refining the RAG pipeline.

How do we verify or audit where the AI got its answer before making a business decision?

There are few ways we can verify and audit the AI response, and should be used as and when required and not taken as a set in stone step.

1. Reference the source

Every answer will come with a citation back to the document the information was retrieved, ideally as specific as to a paragraph, chunk and not just the full document.

2. Confidence Score

The LLM defines a confidence score on the response based on how well supported the answer is.

3. Faithfulness Check

For higher stake projects, we add a 3rd layer. The question, the retrieved chunks and answer goes through a second AI model asking whether the source supports the claim. This adds cost and latency, so is reserved for projects and decisions with high stakes.

As mentioned above, we as developers cannot guarantee 100% accuracy of the response, hence a human review is still advised for high-stake decisions.

Chathuri outlined a flexible, multi-layered approach to auditing AI responses, starting with granular citations back to specific paragraphs or chunks rather than whole documents. She noted that confidence scores provide an immediate indicator of support, while high-stakes decisions can trigger an additional faithfulness check where a second AI model verifies the claim against the source, despite added cost and latency. Ultimately, she emphasized that because automated absolute accuracy cannot be guaranteed, human review remains vital for critical business choices.

What are the biggest hidden costs or friction points when deploying RAG in an enterprise environment?

The obvious costs are the model tokens, embedding costs and vector database hosting.

Preparing and standardizing data before it's vectorized can cost significant human labour, depending on how extensive and messy the source content already is. And most often that data doesn't stay static. Keeping source documents current isn't a one time setup task. Instead it needs an owner and an ongoing process

That is the biggest friction point (or point of failure) as I see it. Unstructured data and stale documentation, both of which are easy to underestimate because they don't show up as a line item the way infrastructure costs do.

If the content needs continuous update, it will also carry knock-on cost where every change requires a re-embed. Left unmanaged, this cost can creep up on you as the document set grows. However we can keep this in check by building incremental re-embedding by re-processing only the content that changed.

We frequently explore these architectural balances when delivering custom software development and conducting upfront discovery workshops to help clients avoid hidden infrastructure creep. You can learn more in our detailed analysis on AI integration cost for mid-sized enterprises.

A newer approach: PageIndex is being used instead of traditional RAG models for certain projects.

Instead of chunking and embedding documents into a vector database, PageIndex builds a hierarchical, table-of-contents-style tree of a document and has the AI reason its way through that tree to find the right section the way a human expert would flip to the right chapter, then section, then page, rather than searching for text that merely looks similar. It skips chunking and vector databases entirely.

It's particularly strong on long, well-structured documents (like financial reports or legal contracts) where precision matters more than speed, but it trades the near-instant lookup of vector search for a slower, reasoning-based retrieval step and it doesn't offer much benefit on short, unstructured content like chat logs or emails.

Ready to transform your scattered enterprise data into an accurate, audit-ready AI system?

At Lizard Global, we engineer grounded, enterprise-grade RAG architectures using custom software development designed to eliminate hallucinations, enforce strict data privacy, and deliver measurable ROI through our impactful works. Don't let bad data hold back your digital transformation.

👉Speak To Our AI Engineering Experts Today