RAG (Retrieval-Augmented Generation) is the most practical AI pattern in 2026. Here is how to build one that actually works.
LLMs have a knowledge cutoff and do not know your private data. RAG fixes both problems: Without RAG: User: "What does our refund policy say about digital goods?" LLM: makes something up With RAG: User: "What does our refund policy say about digital goods?" System: searches your policy docs, finds relevant sections LLM: "According to your policy document, digital goods are non-refundable except..."
User Query | Embedding Model (converts query to vector) | Vector DB Search (finds similar document chunks) | Retrieved Context + Original Query | LLM (generates answer grounded in your data) | Response
from openai import OpenAI
import chromadb
client = OpenAI()
chroma = chromadb.Client()
collection = chroma.create_collection("company-docs")
def chunk_text(text: str, chunk_size: int = 500) -> list[str]:
words = text.split()
chunks = []
for i in range(0, len(words), chunk_size):
chunk = " ".join(words[i:i + chunk_size])
chunks.append(chunk)
return chunks
def embed_and_store(doc_text: str, doc_id: str):
chunks = chunk_text(doc_text)
for i, chunk in enumerate(chunks):
embedding = client.embeddings.create(
input=chunk,
model="text-embedding-3-small"
).data[0].embedding
collection.add(
documents=[chunk],
embeddings=[embedding],
ids=[f"{doc_id}-{i}"]
)def answer_question(user_question: str) -> str:
# Embed the question
query_embedding = client.embeddings.create(
input=user_question,
model="text-embedding-3-small"
).data[0].embedding
# Find relevant chunks
results = collection.query(
query_embeddings=[query_embedding],
n_results=5
)
context = "
".join(results['documents'][0])
# Generate answer
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": "Answer questions based only on the provided context. If the context does not contain the answer, say so."
},
{
"role": "user",
"content": f"Context:
{context}
Question: {user_question}"
}
]
)
return response.choices[0].message.contentChunking strategy matters — Too small loses context. Too large dilutes relevance. 300-500 tokens with 50-token overlap is a good start.
Metadata filtering — Store document source, date, category alongside vectors. Filter before semantic search for better precision.
Reranking — After retrieving top-10 chunks, use a reranker model to pick the best 3. Cohere Rerank or a cross-encoder model.
Evaluation — Build a test set of question-answer pairs. Measure retrieval accuracy and answer quality. RAG quality is measurable.
This pattern powers most enterprise AI deployments in 2026. Learn it well.