A customer types into the chat window: “Can I return an item I bought on sale?” A general-purpose AI model gives a fluent, confident reply, yet it has never read your return policy. If the answer is right, that is luck. This post explains the approach that solves the problem, RAG, without drowning it in technical detail: what it is, how it is built, where it stumbles and where it makes sense to start.
What the problem is and what RAG does
A language model learns from the general text it saw during training. Your price list, delivery times and the campaign rules that changed last month are not in that text. The model does not stay silent, though: it builds a plausible-looking answer from patterns seen at similar companies.
That is exactly where the danger lies. A wrong answer does not look wrong; it arrives in tidy sentences with the same confidence. The model does not pay for it; your support team and your brand do. In an earlier post we asked “Which data holds the right answer?” RAG is the practical response to that question.
RAG stands for “retrieval-augmented generation”: generating an answer with the help of looked-up material. The idea is simple. Before answering, the system searches your own documents for the relevant passages, hands them to the model with the question, and asks the model to answer from those passages only.
Think of a new member of staff. You do not make them memorise everything; you show them the handbook and say “if you are not sure, look it up first, then answer”. An assistant built with RAG works the same way, and it can state under its answer which section of which document it relied on, so your team can check the answer against the source.
A good assistant is not one that knows everything, but one that knows where to look and says so when it finds nothing.
The steps, in order
From the outside there is only a chat window; behind it are six steps that follow one another. The order matters, because each step leans on the quality of the one before.
- 1
Collect and tidy the documents: We gather the documents an answer should rest on, such as the return policy, price list and frequently asked questions. We weed out old versions, turn scanned files into text and make sure every document has an owner.
- 2
Split them into passages: We divide long documents into short passages that each cover one subject. Too short and the context is lost; too long and unrelated information leaks into the answer.
- 3
Make them searchable by meaning: We turn each passage into a numerical summary of its meaning and store it in an index. This is how the question “can I get my money back” finds the return policy passage even though the word return never appears in it.
- 4
Retrieve the relevant passages at question time: When a user asks something, the system searches the index and picks the few passages most relevant to the question.
- 5
Generate an answer that shows its source: The selected passages go to the model along with the question. The model writes the answer from them and states which document it relied on.
- 6
Hand off to a person when unsure: If no relevant passage is found, or the question touches a sensitive subject, the assistant does not guess; it passes the conversation, with the exchange so far, to a member of your team.
If you come across the technical terms
The numerical summary in step three is called an “embedding” in technical language, and the index that stores those summaries a “vector database”. If you see these words in a proposal, this is what they mean: the plumbing that lets text be searched by meaning instead of exact wording.
Document quality comes before the model
The question we hear most at the start of a project is “Which model will we use?” Yet the outcome is usually decided by the documents, not the model. The assistant answers only from the text it is given; if that text is incomplete, scattered or contradictory, even the most capable model carries the same gap and the same contradiction into its answer.
That is why most of the effort goes into putting documents in order: deciding which of three files on the same subject is valid, writing down rules that live only in the team's heads, turning prices trapped in images into text. In our experience a plain setup on tidy documents usually does better than an elaborate setup on messy ones. The model, meanwhile, can be swapped later without touching the documents.
How do we measure whether it works?
“We asked a few questions and it answered well” is not a measurement. Before launch we prepare a test set of real questions, taken from your support records, emails and WhatsApp conversations, in the customers' own words. Next to each question we write the correct answer and the document it lives in.
Then we look at two things separately: did the system find the right passage, and did the model draw the right answer from it? The two have different fixes. The set also contains questions whose answers are not in the documents; there, we expect the assistant to say it does not know and hand the conversation over. Whenever a document, the splitting or the model changes, we run the same set again, so we see from results, not guesswork, whether fixing one thing has broken another.
Where does it go wrong?
RAG reduces the model's tendency to make things up, but it does not run flawlessly by itself. Most of the problems we meet fall under three headings.
- Outdated documents: The price has changed but the list in the index is still the old one; the assistant quotes the old price and cites a source for it. The fix is to decide up front who updates the documents and how often, and to tie the index to those updates.
- Conflicting versions: Two versions of the same rule sit in two separate files, and the same question gets different answers on different days. The fix is to leave a single valid document for each subject.
- Questions outside the documents: The customer asks something no document answers. A poorly built assistant fills the gap with a guess; a well built one says it does not know and hands the conversation to a person.
There are also tasks where an error is expensive, such as approving a return, quoting a price or interpreting a contract clause. For these the assistant only prepares a draft; we leave the final word to a person's approval.
Data, privacy and monthly cost
Connecting your documents to an assistant means deciding which data goes where. We put this in writing at the start of the project: which documents go into the index, which model provider the questions and retrieved passages are sent to, and where conversation records are kept and for how long. We mask sensitive fields such as names, phone numbers and card details before anything reaches the model. Public terms and internal correspondence meant only for the team should not share an index.
On the cost side there are two items: a one-off build and a running cost that depends on usage. Each answer carries a small fee to the model provider, and it grows with the number of questions and the length of the text handed to the model. So instead of quoting a fixed figure, we estimate the monthly cost from expected volume and track real usage after launch.
RAG or training the model?
Two approaches are often confused. Retraining a model on your own data, known technically as fine-tuning, means permanently changing its behaviour through examples. RAG leaves the model untouched; it puts the right document in front of it at the moment of answering. One is like teaching an employee a style and a set of habits, the other like placing the current handbook on their desk.
- When RAG is appropriate: The information changes often, the answer needs to show its source, and the content lives in documents, such as prices, terms or procedures. When the document is updated the answer is updated too; no retraining is needed.
- When training is appropriate: The problem is behaviour, not missing knowledge. The model cannot keep to a particular writing style, a fixed output format or a narrow classification task consistently enough, and you have enough good examples to teach it.
- Both together: The two are not necessarily alternatives, and some projects use them side by side. Even so, for an assistant that answers from documents the starting point is almost always RAG.
A trained model does not memorise your price list reliably, nor does it forget the old figure by itself when the price changes. Knowledge that has to stay current belongs in a document, not inside the model.
Starting small: one document set, one channel
In the first version we do not try to load the whole company's knowledge into the assistant. We pick one document set, for example return and delivery terms, and one channel, for example your WhatsApp line or the chat window on your website. A narrow scope keeps the test set small and meaningful, and makes it easier to see where errors come from.
A suggestion for the first week
Ask your support team to collect the twenty or thirty questions they have received most often recently, exactly as customers wrote them. Then mark which document holds the answer to each. Questions with no answer in any document show which documents need writing first, and the list doubles as the core of your first test set.
Once the assistant works measurably well in this narrow area, widening the scope is easy: a new document set, a second channel, an automation step connected to your order system. That is the essence of RAG: instead of trying to make the model cleverer, give it the right document, show where the answer came from, and leave room for a person wherever the assistant is unsure.
