Choose a RAG Development Agency That Tests Retrieval First

A RAG development agency should start with one risky question

Start with a question whose answer matters. A support lead may need to know whether a customer can cancel after renewal. The answer depends on the current contract, the customer's plan and a regional policy, not on every file in a shared drive.

Ask the agency to trace that question from the user to the approved source. The first map should name who asks, which documents are eligible, which permissions apply, how current the answer must be and what happens when the evidence conflicts.

That map prevents a common mistake: indexing a large document folder before anyone defines success. A product discovery workshop can narrow the first workflow, but the output must be a testable decision, not a list of possible chatbot features.

Write a retrieval contract before choosing the interface

A retrieval contract states what the system may search and what a useful result contains. It should cover source ownership, user access, document freshness, citations, refusal behavior and the maximum consequence of a wrong answer.

For the cancellation question, the contract might allow approved policy pages and the signed customer agreement, require a citation to each, and refuse when the two sources disagree. It should not silently fill the gap from general model knowledge.

OpenAI's Retrieval guide describes vector stores that return chunks, file origins and similarity scores. Its File Search guide adds keyword and semantic search, metadata filters and file citations. Those are useful mechanisms, but the agency still has to define which metadata and thresholds express your product rules.

Test retrieval before testing the answer

A fluent answer can hide a retrieval failure. Separate the system into two questions: did it find the required evidence, and did it use that evidence correctly?

Build an evaluation set from real work. Include common wording, shorthand, misspellings, older terms and questions that should return nothing. For each item, record the expected source, acceptable alternatives, forbidden sources and whether the system should abstain.

Suppose staff call a product “Flex” while the policy uses its legal name. Semantic search may connect the terms even when they share few words, which OpenAI documents as a retrieval benefit. The test still needs to show that the correct current policy ranks above an archived launch note. Ask for retrieval results and scores, not only the final paragraph.

Permissions must survive the trip into the index

A document that requires authentication in its original system can become broadly visible after ingestion if access metadata is lost. The RAG design must carry the user's authorization into every retrieval request.

Imagine HR and sales policies share one knowledge base. A sales manager asks about parental leave and retrieves a confidential draft intended for HR. The model did not invent the leak. The retrieval layer served a file that the user should never have seen.

Require tests for role changes, revoked access, shared links and deleted documents. Metadata filtering can restrict results, but filters are only as trustworthy as the identity and synchronization process behind them. The OWASP RAG Security Cheat Sheet recommends access controls, data validation and continuous monitoring across the pipeline.

Treat retrieved content as untrusted input

RAG does not make prompt injection disappear. A retrieved document can contain instructions that try to override the product's rules, reveal data or trigger tools.

A supplier PDF might include hidden text telling the assistant to ignore its task and send stored credentials. The useful business content and the hostile instruction arrive through the same retrieval path.

OWASP's Prompt Injection guidance says retrieval-augmented generation does not fully mitigate the risk. Keep tool permissions narrow, separate instructions from retrieved material, validate outputs before side effects and test hostile documents. A read-only policy assistant has a smaller failure surface than an assistant that can email, edit records or approve refunds.

Budget for freshness, refusal and operating ownership

The index changes after launch. Documents are revised, access is revoked and user language drifts. The proposal should price the operating loop, not just the first ingestion.

For a policy assistant, name how quickly an approved revision reaches search, how stale copies are removed and who investigates a missing citation. Record the retrieval query, eligible source set, returned chunks and final citations without logging sensitive content indiscriminately.

There is a real tradeoff between quality and cost. Returning fewer chunks can reduce latency and token use, while OpenAI warns that limiting results can reduce answer quality. Test that tradeoff on your evaluation set. An AI product audit can examine an existing implementation; an AI product requirements document can make the new operating rules explicit before procurement.

Reject the demo that answers only curated questions

A polished prototype is useful when it exposes the difficult cases. It is weak evidence when the agency controls every question and document.

Bring unseen questions, conflicting versions, a revoked user and a document with misleading instructions. Ask the agency to show the raw retrieval results, not to explain the intended architecture after a failure.

The serious objection is that this evaluation work slows an early prototype. It does. For a low-risk internal idea, a small experiment may be enough. But when answers affect customers, money, employment or regulated decisions, the test set is part of the product. The broader AI product development partner guide helps compare delivery capability beyond retrieval.

Use this checklist before hiring a RAG development agency

The buying evidence should fit on one page. Check these items before comparing model names or chat animations.

• One valuable question and the consequence of a wrong answer are explicit.
• Approved sources, owners, exclusions and freshness expectations are named.
• User permissions are enforced at retrieval time and tested after access changes.
• The evaluation set includes expected sources, refusals, conflicts and hostile content.
• Retrieval quality is measured separately from answer quality.
• Citations let a user inspect the supporting source.
• Ingestion, monitoring, incident response and evaluation have operating owners.

Bring Seuil one question your knowledge system must answer safely

Send Seuil one real question, the sources an authorized person would use and the consequence of getting it wrong. That is enough to define the retrieval contract, the first evaluation set and the smallest credible build without pretending a chatbot demo proves the product.