An explanation of retrieval-augmented generation (RAG), how it gives AI access to external information and why it can reduce hallucinations.
If you have spent much time using AI tools, you have probably discovered that they have an awkward habit. They can give an answer that sounds completely convincing while being completely wrong.
There is another problem. An AI model does not automatically know about your private information. Ask it about your company's internal procedures, documentation for a private project or the configuration of your home network and, unless you provide that information, it has nothing reliable to work from.
One way of dealing with both problems is retrieval-augmented generation, usually shortened to RAG.
The name makes it sound considerably more complicated than the basic idea actually is.
Think of RAG as an open-book test
A useful way to understand RAG is to think about the difference between a closed-book and open-book exam.
A normal AI conversation is somewhat like a closed-book test. The model has what it learnt during training, along with whatever information you provide in the current conversation. If the information needed to answer a question is not available, the model may still attempt to produce an answer.
That is where hallucinations can occur. In AI terminology, a hallucination is an answer containing information generated by the model that is incorrect or unsupported, despite sometimes sounding perfectly plausible.
RAG turns the process into something closer to an open-book test.
Before asking the AI to answer, the RAG system searches a collection of information that it has been given access to. This might include company documentation, manuals, policies, websites, project files or other sources.
It finds the information most relevant to your question and supplies that information to the AI along with the question. The AI can then use those retrieved documents as reference material when producing its answer.
What happens when you ask a question?
At its simplest, RAG involves three steps.
- Retrieve. The system searches its available information and finds the passages most relevant to your question.
- Augment. Those passages are added to the information supplied to the AI model.
- Generate. The model uses your question and the retrieved information to produce its answer.
The documents are usually prepared for searching beforehand by breaking them into smaller sections, often called chunks. Rather than handing the AI an entire 200-page manual, the system might retrieve only the few paragraphs that appear relevant to the question.
Many RAG systems can also search by meaning rather than relying entirely on exact words. For example, a question about "resetting my password" could still locate instructions headed "account credential recovery" even though the wording is different.
You do not need to understand the mathematics behind that process to understand RAG. The important part is that the system tries to find useful information first and gives it to the AI before asking for an answer.
Why not just train the AI on your documents?
Training and fine-tuning are different from RAG.
Fine-tuning changes a model to make it better suited to particular tasks, behaviour or types of output. RAG leaves the underlying model alone and instead gives it access to relevant information when that information is needed.
That distinction matters when the information changes regularly.
Suppose a company changes its leave policy. With a RAG system, the organisation can update the source document and update the searchable information. Future questions can then retrieve the new policy without changing the AI model itself.
The same principle works for product manuals, technical documentation, price lists, support information and other material that changes over time.
RAG can also show where an answer came from
Another useful feature of a well-designed RAG system is that it knows which documents it retrieved.
That means an application can provide references or citations with an answer, allowing the user to check the original information rather than simply trusting whatever the AI says.
This is particularly useful when an answer is based on company policies, technical documentation or other information where getting the details right matters.
Access controls can also be applied to the retrieval system. An employee might be allowed to search general company procedures, for example, while confidential finance or personnel documents remain unavailable. This security has to be designed into the system; RAG does not automatically make private information secure.
RAG does not make AI infallible
RAG is sometimes described as a way of stopping AI hallucinations, but that overstates what it can do.
It can significantly improve the information available to the model, but the answer is only as good as the information retrieved. If the source documents are wrong, out of date or incomplete, the resulting answer can also be wrong. The search can retrieve the wrong passage, or the model can misinterpret otherwise correct information.
A better way to describe RAG is that it grounds an AI's answer in information retrieved from known sources.
Instead of expecting the model to know everything, RAG gives it somewhere to look.
That relatively simple idea is what makes it possible to turn a general-purpose AI model into something that can answer questions about your documentation, your organisation or another specialised collection of information, without having to teach all of that information to the model itself.
COMMENTS