RAG: How to Make AI Answer from Your Company's Documents

Welcome to Level 4: Expert. You already know how to use AI, build assistants, automate and call AI from Python. At this level you'll build solutions businesses pay good money for: assistants that answer from your own documents, customer service chatbots and tools built by coding with AI. We'll finish with how to turn all this into income. We start with the most widely used technique in business AI: RAG.

The problem: AI doesn't know your company

A language model learned from public text up to a certain date (lesson 2). It doesn't know your return policy, your new location's hours or what your vendor contract says. If you ask, it may invent a convincing answer. There are three ways to give it that knowledge:

Many "project" and knowledge-base tools use RAG under the hood. Understanding how it works helps you configure them better and, if you know Python, build your own.

How RAG works in four steps

1. Step 1 · onceSplitDocuments are cut into chunks of one or a few paragraphs.
2. Step 2 · onceIndexEach chunk is turned into a representation that makes it fast to search.
3. Step 3 · every questionRetrieveWhen a question arrives, the most similar chunks are found.
4. Step 4 · every questionGenerateThe model receives the question and those chunks, with instructions to answer only from them and cite the source.

What are embeddings?

An embedding is a list of numbers (a vector) that represents the meaning of a text. Texts with similar meanings get vectors that sit close together, even if they share no words: "can I send back a coffee?" and "return policy" end up near each other. AI providers and several open-source libraries offer embedding models, and vector databases store them and search millions of them in milliseconds.

Build it in Python

Let's build the assistant for Ridgeline Coffee, the illustrative Portland coffee company from earlier lessons, which has two cafés. So you can run the search part with no keys and no cost, we'll use a classic keyword-search technique, TF-IDF, included in scikit-learn (pip install scikit-learn). Then you'll see why embeddings beat it.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

documents = [
    {"source": "return-policy.txt",
     "text": "We accept returns of unopened packaged products within 15 days of purchase "
             "with a receipt. Prepared drinks are non-refundable."},
    {"source": "hours.txt",
     "text": "The Pearl District cafe is open Monday through Saturday from 7 a.m. to 9 p.m. "
             "and Sundays from 8 a.m. to 2 p.m. The Hawthorne cafe is closed on Sundays."},
    {"source": "delivery.txt",
     "text": "We deliver within a 3-mile radius. The minimum order is $15 and the delivery "
             "fee is $4; delivery is free on orders of $40 or more."},
    {"source": "loyalty-program.txt",
     "text": "With the Ridgeline card you earn one stamp per drink. Collect 9 stamps and "
             "your 10th drink is free. Stamps expire after 6 months."},
    {"source": "allergens.txt",
     "text": "All pastries may contain traces of tree nuts, gluten and dairy. We offer "
             "almond and oat milk as dairy-free alternatives."},
]

vectorizer = TfidfVectorizer(stop_words="english")
matrix = vectorizer.fit_transform([d["text"] for d in documents])

def search(question, k=2, minimum=0.1):
    sim = cosine_similarity(vectorizer.transform([question]), matrix)[0]
    order = sim.argsort()[::-1][:k]
    return [documents[i] | {"score": round(float(sim[i]), 2)}
            for i in order if sim[i] >= minimum]

for q in ["How much is home delivery?",
          "Is the Hawthorne cafe open on Sunday?",
          "Can I return a coffee I didn't like?"]:
    print(q, [(d["source"], d["score"]) for d in search(q)])

How much is home delivery? [('delivery.txt', 0.55)]
Is the Hawthorne cafe open on Sunday? [('hours.txt', 0.6)]
Can I return a coffee I didn't like? []

The first two questions find the right document. The third finds nothing: the question says "return," the document says "returns," and to a keyword search those are different words (and "coffee" doesn't appear in any document). Look closely at the second one too: "Sunday" didn't match "Sundays" either; it was "Hawthorne," "cafe" and "open" that carried it. That's exactly the problem embeddings solve, because they compare meanings. In a real system you'd replace the TF-IDF vectorizer with an embedding model and keep the rest of the code. Still, notice something good: since no chunk cleared the minimum, the system knows it doesn't know, instead of sending an irrelevant document.

Now try the reverse: search("Do you accept returns?") does find return-policy.txt (score 0.4), because now the words match. Same intent, different wording, different result: that's how fragile keyword search is.

The prompt that prevents made-up answers

def build_prompt(question):
    chunks = search(question)
    if not chunks:
        return None          # nothing relevant: hand off to a person
    context = "\n\n".join(f"[{c['source']}]\n{c['text']}" for c in chunks)
    return (
        "Answer the customer's question using ONLY the information in the documents.\n"
        "If the answer isn't in the documents, say you don't know and offer to connect "
        "them with a person.\n"
        "At the end, put the source you used in square brackets.\n\n"
        f"<documents>\n{context}\n</documents>\n\n"
        f"Question: {question}"
    )

And the call to the model, just like in lesson 10:

import anthropic

client = anthropic.Anthropic()      # reads the key from the environment variable

def answer(question):
    prompt = build_prompt(question)
    if prompt is None:
        return "I don't have that information. Would you like me to connect you with someone on our team?"
    r = client.messages.create(
        model="claude-sonnet-5-5",   # check the current models in the docs
        max_tokens=400,
        messages=[{"role": "user", "content": prompt}],
    )
    return r.content[0].text

print(answer("How much is home delivery?"))

A typical answer: "Delivery costs $4 within a 3-mile radius, with a $15 minimum order. It's free on orders of $40 or more. [delivery.txt]". And for the returns question, the code doesn't even call the model: it goes straight to the handoff message, which costs nothing and invents nothing.

What makes a RAG system good (or bad)

No code? Test the idea in your assistant

You can simulate RAG in any chat by pasting the "retrieved chunks" yourself. It's a cheap way to test the instructions before investing in development:

You're the assistant for Ridgeline Coffee. Answer ONLY from the
documents. If the answer isn't there, say "I don't have that
information" and offer to connect them with a person. Cite the
source in square brackets.
<documents>
[delivery.txt] We deliver within a 3-mile radius. The minimum order
is $15 and the delivery fee is $4; delivery is free on orders of
$40 or more.
</documents>
Question: Do you deliver to St. Johns, and how much would I pay on a $25 order?

Expected result: "On a $25 order, delivery is $4 [delivery.txt]. I don't have information on whether St. Johns is within our 3-mile radius; I can connect you with someone to confirm." If it instead says yes, you deliver to St. Johns, your instructions aren't enough: the document doesn't say which neighborhoods are covered.

documents += [
    {"source": "payment-methods.txt",
     "text": "We accept cash, all major credit and debit cards, Apple Pay, Google Pay "
             "and Venmo. Gift cards can be used at both cafes."},
    {"source": "event-bookings.txt",
     "text": "You can reserve the Hawthorne back room for private events of up to 20 "
             "guests. Bookings require a $100 deposit and at least 7 days' notice."},
]
matrix = vectorizer.fit_transform([d["text"] for d in documents])   # re-index

tests = [
    ("How much is delivery?", "delivery.txt"),
    ("What time does the Pearl District cafe open on Sundays?", "hours.txt"),
    ("Do the stamps on my card expire?", "loyalty-program.txt"),
    ("Do you have oat milk?", "allergens.txt"),
    ("Can I pay with Venmo or a credit card?", "payment-methods.txt"),
    ("Can I rent a space for a birthday party?", "event-bookings.txt"),
    ("Do you sell shares in the company?", None),      # not covered: hand off
    ("What's the Wi-Fi password?", None),
]

def evaluate(tests):
    hits = 0
    for question, expected in tests:
        r = search(question)
        got = r[0]["source"] if r else None
        ok = got == expected
        hits += ok
        print("✓" if ok else "✗", question, "→", got)
    print(f"Accuracy: {hits}/{len(tests)} ({hits / len(tests):.0%})")

evaluate(tests)

Summary

• AI doesn't know your documents: paste them, use projects with files or build a RAG system.
• RAG splits, indexes, retrieves the relevant chunks and generates the answer from them alone.
• Embeddings compare meanings; keyword search fails on synonyms and word variants.
• Instruct the model to answer only from the documents, admit when it doesn't know and cite the source; a score threshold lets the system say "I don't know."
• Current documents, access permissions, privacy and an evaluation set make RAG trustworthy.

Next lesson: AI Customer Service Chatbot: Website, SMS and WhatsApp, Step by Step. You'll use these pieces to design a complete chatbot: what to automate, tone, human handoff and metrics.