What is RAG?

Glossary

RAG, retrieval-augmented generation, makes an AI model look up relevant passages from your own documents first and then answer using them, cutting made-up answers.

RAG stands for retrieval-augmented generation. Before an AI model answers a question, the system searches a set of documents you control, pulls out the most relevant passages, and hands them to the model along with the question. The model then writes its answer from that material instead of relying only on what it absorbed in training.

The retrieval step and the generation step

Setup happens once. Your documents, such as help articles, policies or product sheets, are split into chunks of a few paragraphs. Each chunk is turned into an embedding, a list of numbers that captures its meaning, and stored in a vector database or search index.

At question time, the question is turned into an embedding too, and the system finds the chunks whose numbers are closest. Many setups also run ordinary keyword search and combine the results. The top chunks are inserted into the prompt with an instruction like "answer using only these sources". The LLM writes the reply, often citing which chunk it used. The approach was named in a 2020 research paper from Facebook AI Research.

A worked example: a returns question

An outdoor gear shop adds a help chatbot to its site. A customer asks, "Can I return a tent I've used once?" Retrieval finds the paragraph of the returns policy about used items and the paragraph on the 60-day window the shop actually offers. The model answers from those two passages. Without RAG, the model would guess from typical store policies and might promise something the shop does not offer.

Why RAG matters for a small business

RAG is how you make an AI assistant speak for your business specifically. It keeps answers current, because updating the documents updates the answers with no retraining, and it lets you show sources. If you only use AI for drafting marketing copy, you do not need it. If you want a support bot or an internal assistant over your own files, you almost certainly do.

Common RAG mistakes

Feeding it messy or outdated documents. The model repeats whatever it retrieves, including last year's prices.
Chunks that are too small or too large. Tiny chunks lose context; huge ones bury the useful sentence.
No fallback for missing answers. Instruct the model to say it does not know and offer a contact option when nothing relevant is found.
Skipping evaluation. Keep a list of real customer questions and check the answers whenever documents change.