Agentic RAG vs Classic RAG: A Measured Comparison on a Small Knowledge Base

What the two setups do

Classic RAG takes the user's question, searches once, puts the top results into the prompt and asks the model to answer from them. Agentic RAG gives the model the search function as a tool (see how function calling works) and lets it call the tool as often as it needs, with its own queries, before it answers. The search step is the same in both; what changes is who writes the queries and how many times the search runs.

The test

To make sure the model could not know the answers from training, I invented a small company, "Larkspur Robotics", and wrote 24 one-sentence passages about its projects, people, plants and offices. Some passages are distractors (for example, other managers who "lead" other things). Search is a plain BM25 function in Python that returns the top 3 passages: no API call, no embeddings, no stemming. The model was gemini-3.1-flash-lite, temperature 0, run on 9 October 2026 with a free-tier key.

The "calls" column counts model calls: a question that needs one search costs two (one to ask for the search, one to answer). Output tokens, including the model's reasoning tokens, were 15 for classic RAG and 209 for the loop.

What each setup did on the two-step questions

For question 3, classic RAG searched with the question text and got three passages, none of which said who leads Project Kestrel: two were about other managers who "lead" meetings and drills, and one was about where Kestrel is tested. The model answered "NOT FOUND", which is the correct response to a context that lacks the fact. The agentic loop issued its own queries, one per fact (each search returned three passages; only the relevant one is shown):

search("who leads Project Kestrel")          -> Project Kestrel is led by Dev Anand.
search("who is the manager of Dev Anand")   -> Dev Anand reports to Ingrid Solberg, the Chief Technology Officer.
answer: Ingrid Solberg

Question 4 added a third search ("where is Ingrid Solberg's office located") and returned Oslo. Each query was short and about a single entity, which is what BM25 handles well.

The loop, in code

TOOL = {"functionDeclarations": [{
    "name": "search",
    "description": "Search the company knowledge base. Returns up to 3 short passages.",
    "parameters": {"type": "OBJECT", "properties": {"query": {"type": "STRING"}}, "required": ["query"]}}]}

SYSTEM = ("Answer the question using the search tool. Search as many times as you need, one fact at a time. "
          "Answer only from what the search results say. If you cannot find it, reply exactly: NOT FOUND.")

def agentic(question, call_model, search, max_steps=6):
    contents = [{"role": "user", "parts": [{"text": question}]}]
    for _ in range(max_steps):
        reply = call_model(contents=contents, tools=[TOOL], system=SYSTEM)   # one generateContent request
        parts = reply["candidates"][0]["content"]["parts"]
        calls = [p["functionCall"] for p in parts if "functionCall" in p]
        if not calls:                                   # no tool call: this is the answer
            return "".join(p.get("text", "") for p in parts)
        contents.append({"role": "model", "parts": parts})
        results = [{"functionResponse": {"name": "search",
                    "response": {"results": search(c["args"]["query"])}}} for c in calls]
        contents.append({"role": "user", "parts": results})
    return "(step limit reached)"

The step limit matters: without it, a model that never finds the fact can keep searching. In my four questions the loop always stopped on its own, well under the limit of six.

What this does and does not show

• The cost is real and visible. The loop used 2.75 times as many model calls and 5.4 times as many input tokens as classic RAG over these questions, because every step re-sends the conversation so far. On the single-fact questions it bought nothing: same answers, about three times the input tokens.
• Part of classic RAG's failure was the search function. BM25 without stemming matched "leads" in the question against the two unrelated "who leads ..." passages, and did not match "led" in the right one. A dense-embedding search, or a stemmer, would probably rank the right passage higher and might let classic RAG answer question 3. I did not test that. The loop worked around the weak search by writing better queries.
• Four questions prove little. One model, one run each, a tiny invented corpus, and success judged by whether the expected string appears in the answer. Expect different numbers on your own data.
• Models not tested. Larger models might plan more searches in fewer calls, or fewer; I only ran the model named above.

When I would use which

Use classic RAG when most questions are answered by one passage and you care about latency and cost. Add the loop when questions chain facts together or when a single search often misses, and put a step limit and a token budget on it. A cheaper step before either is to improve the search itself (stemming, embeddings, query rewriting) and measure again.

Sources and Further Reading

• Function calling with the Gemini API — Google AI for Developers
• Okapi BM25 — Wikipedia (the ranking function used for the search step)