Building a RAG System: Architecture, Costs and Common Mistakes

A RAG system (retrieval-augmented generation) is an AI application that first retrieves the relevant passages from your own documents for every question and then lets the language model answer only on that basis, citing its sources. This lets a chatbot or AI agent use company knowledge without training a model on your data. Whether the system turns out well rarely depends on the model. It depends on the architecture around it: document preparation, search, permissions and testing.

At TAISC in Karlsruhe, Germany, we build RAG systems for mid-sized companies, from internal knowledge chatbots to error analysis in IT support, and in September 2026 we ran a two-day advanced training on RAG. This article explains how a RAG system is built, what it costs, when you do not need one at all and which mistakes sink most projects.

What is a RAG system?

A large language model (LLM) knows the public knowledge from its training, but not your manuals, policies, contracts or support tickets. Ask it about them anyway and it guesses, convincingly. These hallucinations are the main reason why plain chat access quickly hits its limits in a company.

RAG solves this by splitting the work. A search component, the retrieval step, finds the relevant passages in your content. The language model receives those passages together with the question and writes the answer, the generation step. The term comes from a 2020 research paper by Lewis and colleagues at Facebook AI Research. Today RAG is the standard pattern whenever AI needs to work with company knowledge, and it sits behind many “chat with your documents” features of the big vendors.

Three properties make RAG attractive for businesses:

• Freshness: when a document changes, it is re-indexed. No model has to be retrained.
• Traceability: every answer can point to the passages it is based on. Users can verify instead of trust.
• Control: the data stays in your system. You decide which sources are connected and who may see what.

These are exactly the points that slow down AI adoption in Germany according to the digital association Bitkom. In its survey of 604 companies (Bitkom Research 2025, report published February 2026), 48 percent name data protection requirements as an obstacle, 39 percent worry that company data could end up in the wrong hands and 38 percent criticise the lack of traceability of results. A well-built RAG system addresses all three.

How does a RAG pipeline work?

A RAG pipeline has two parts: indexing, which runs once and on every change, and querying, which runs for every question.

Indexing:

1. Load and parse. Documents are pulled from the source systems and converted into clean text, including tables, headings and page numbers. For PDFs this is the most underestimated step.
2. Split. The text is cut into sections, known as chunking. Good chunks are self-contained and carry their context, for example the path of headings above them.
3. Embed. Each chunk is turned into a vector of numbers that captures its meaning, an embedding. Similar content ends up close together even when it uses different words.
4. Store. Vectors, text and metadata such as source, date and permissions go into a vector database.

Querying:

1. The question is embedded as well, often rewritten or split into sub-questions first.
2. Search returns the most similar chunks, ideally combining vector search with classic keyword search.
3. A reranker re-orders the candidates by actual relevance and keeps only the best.
4. The language model receives the question and the chunks and answers with sources, or says honestly that the documents do not answer the question.

We first learned how embeddings behave in practice with our own AI News Digest. The agent embeds a week’s newsletters and groups texts above a fixed cosine similarity threshold into one topic, so three newsletters about the same announcement become one entry. It is the same principle a RAG system uses to find the right passages for a question. It also showed us why we do not let the language model group content directly: the results are harder to compare and harder to check.

RAG architecture: the building blocks

The table shows what the Claude Code source code, accidentally published in March 2026, confirmed for coding agents too: a successful AI system consists mostly of conventional software. For Claude Code it was roughly half a million lines of TypeScript for context management, error handling, tools and orchestration around the model. A RAG system is no different. The language model is one row in the table; the other seven decide the quality.

RAG, fine-tuning or long context?

Not every knowledge problem needs a RAG system. The three common approaches compared:

Long context means simply putting the whole knowledge base into the prompt. Anthropic explicitly recommends this for collections below roughly 200,000 tokens, about 500 pages. With prompt caching it stays affordable, and every failure mode of search disappears. For a product manual or a set of policies this is often the better solution.

Fine-tuning changes the model itself. It is good for teaching a model a style, an output format or domain language, but poor at reliably memorising facts. For company knowledge that changes and must be backed by sources, it is almost never the first choice.

RAG is the right call as soon as the collection is too large for the context window, changes regularly, comes from several sources or has different access rights. That is almost always the case for intranets, technical documentation, support knowledge and contract archives.

What does a RAG system cost?

The good news first: the model is rarely the cost driver. Embeddings are extremely cheap. OpenAI’s widely used text-embedding-3-small model costs USD 0.02 per million tokens according to common price trackers (as of October 2026). A knowledge base of 10,000 pages can be fully indexed once for well under one euro.

Running costs per question are manageable too. For Logify, our AI log file analysis for IT support, which searches about 500 MB of technical documentation via RAG, model costs are around one US dollar per 100 queries. Depending on the model, the amount of context passed in and the number of checking steps, this can vary considerably in either direction.

The real budget goes into three other items:

• Data preparation: finding and cleaning documents, removing duplicates and outdated versions, making scans and tables readable. In many projects this is the largest single item.
• Evaluation: building a test set of real questions with verified answers and measuring every change against it. Without it, every optimisation is a gut feeling.
• Integration and operations: connecting source systems, permission checks, re-indexing on change, monitoring and updates when models change.

As a rough guide (estimate): a proof of concept on a well-defined document set is feasible within a few weeks. A production system with several sources, permissions, evaluation and operations is more of a multi-month project. What needs to happen once it is live is covered in our article on hypercare.

Use the following check to get a rough idea of which architecture fits your collection and what the model costs add up to:

Seven common mistakes when building a RAG system

Most RAG prototypes answer ten questions well in the demo and then fail at the eleventh. The causes are almost always the same.

1. Bad documents in, bad answers out

A RAG system can only find what is cleanly indexed. Word files scattered across drives, PDFs with tables as images, five versions of the same manual: all of this lands unfiltered in the index and confuses the search. That is why a lot of time in our projects goes into preparation before the first chunk is created. It was the same with IntranetGPT, a RAG chatbot that makes scattered intranet and policy knowledge available via chat and, according to the client’s internal communication, cut search times by up to 70 percent.

2. Chunks without context

A section saying “The deadline is 14 days” is useless if the system does not know which contract and which chapter it comes from. In the labs of our advanced RAG training, the top-1 hit rate (Hit@1) on difficult questions rose from 0.692 to 0.846 just because every chunk carried its heading path. That was measured on the course dataset; the direction transfers, the exact number does not.

3. Vector search only

Vector search understands meaning but is weak with exact terms such as part numbers, error codes or legal sections. Hybrid search combines it with classic keyword search (BM25). For its “contextual retrieval” method, Anthropic measured in September 2024 that context-enriched embeddings plus keyword search reduced failed retrievals by 49 percent, and by 67 percent with reranking added. These figures come from Anthropic’s own benchmarks and vary noticeably by type of data.

4. No reranking

Search returns the 20 to 50 most similar chunks, but the language model needs the best three to eight. A reranker scores each candidate again specifically against the question. In our training labs, Hit@1 rose from 0.917 to 0.950 as a result. The price: latency grew from 9 to 222 milliseconds. Whether that is worth it is a trade-off you should measure, not guess.

5. No permissions in retrieval

The most dangerous mistake: a chatbot that quotes the executive pay policy to every employee because all documents sit in one index. Permissions must be attached to every chunk as metadata and filtered on every search, before the model sees the text. Retrofitting this cleanly is very hard.

6. No test set

Without a collection of real questions with verified answers, you cannot tell whether a change makes the system better or worse. With a test set you check every change to chunking, model or prompt automatically, ideally as a regression test in your CI/CD pipeline. How to build good test data is covered in our article on ground truth data in RAG systems; background on the method is in the glossary under evaluation.

7. No permitted non-answer

A model that must always answer will also answer when the retrieved passages do not fit. Good RAG systems have a clear rule: if the sources do not answer the question, the system says so and refers to a human. For answers with consequences, a human approval step belongs in the process as well.

How to approach building a RAG system

Across our projects, a six-step approach has proven itself:

1. Collect the use case and questions. Which questions should be answered, by whom, how often? 50 to 100 real questions from daily work are the best starting point and at the same time the core of your later test set.
2. Clarify sources and permissions. Which documents are authoritative, where do they live, who may see what? Without these answers, no code should be written.
3. Check the size. If everything fits into the context window, start with long context. Only build the RAG pipeline if it does not.
4. Start simple, then measure. Clean parsing, chunks with context, hybrid search. Then measure against the test set where the system fails: at finding or at phrasing.
5. Improve where it matters. Reranking, query rewriting or a different embedding model only where measurement shows a bottleneck.
6. Plan operations. Re-indexing on change, monitoring of answers, cost control and a channel for reporting wrong answers.

For agents that should not only read documents but also work with systems, RAG is often combined with tools via the Model Context Protocol. The agent then searches the knowledge base and queries live data at the same time.

RAG chatbots and data protection: what to consider

A RAG system often processes personal data, for example names in meeting notes, emails or tickets. In the EU this means the usual GDPR obligations apply: a legal basis, a data processing agreement with the model provider, a deletion concept and access control. One advantage of RAG over fine-tuning: deleted documents also disappear from the answers with the next re-index, because no knowledge is stored in the model itself.

For sensitive collections there are two routes. Either a European data centre with clear contracts with the model provider, or a local LLM on your own infrastructure. In between there is a middle route, which we took with Logify: since April 2026 the application and indexing have run on-premise on the client’s infrastructure, while the language model itself is connected via the OpenAI API. The architecture allows the model to be swapped for a local one later. Local models are good enough for many RAG tasks today, because they only need to phrase what the search has found.

For a neutral overview in German, the Fraunhofer IAO study “Unternehmensdokumente erschließen mit RAG” from March 2026 is freely available.

Conclusion

A RAG system is the most direct way to let AI work with your company’s knowledge: traceable, and without training your data into a model. The technology is mature and model costs are low. Success or failure is decided by data quality, a search that also finds exact terms, permissions at chunk level and a test set that makes every change measurable. Before you build, also check honestly whether your collection simply fits into the context window.

Frequently asked questions

What is a RAG system?

A RAG system (retrieval-augmented generation) is an AI application that first searches your own documents for passages relevant to each question and then lets the language model answer only on that basis, usually citing sources. This lets AI use company knowledge without training a model on the data.

What is the difference between an LLM and RAG?

An LLM is the language model itself and only knows its training data. RAG is an architecture around the LLM: a search component supplies the model with relevant passages from your data for every question. The model writes the answer; the facts come from your documents.

Is ChatGPT a RAG system?

ChatGPT itself is primarily a chat built on a language model. Features such as web search, uploaded files or connected knowledge sources, however, follow the RAG principle: they search for matching content and pass it to the model as context. Your own RAG system additionally gives you control over sources, permissions and quality.

How much does a RAG system cost?

Model costs are usually low: embeddings cost fractions of a cent per page, and a query is often around one cent. Most of the budget goes into data preparation, evaluation, integration and operations. By our estimate, a proof of concept is feasible within a few weeks, while a production system with permissions and monitoring is more of a multi-month project.

Do I need a vector database for RAG?

For production systems usually yes, because it stores vectors, text and metadata such as permissions together and searches them quickly. For small prototypes a simple index is often enough. Common choices are Qdrant, pgvector as a PostgreSQL extension and ChromaDB. More important than the product is support for hybrid search and metadata filters.

Can we build a RAG system ourselves?

Technically skilled teams can build a prototype with frameworks such as LangChain or LlamaIndex within days. Moving to production is the hard part: clean data preparation, permissions in retrieval, a test set for measuring quality and monitoring. That is exactly where many in-house builds fail.

RAG or fine-tuning: which is better?

For company knowledge that changes and must be backed by sources, RAG is almost always the better choice, because new documents are available without training and answers cite sources. Fine-tuning suits style, format or domain language. For small knowledge bases up to around 500 pages, long context without RAG may be enough.

Do you want to build a RAG system on your own data or take an existing prototype into production? We are happy to work out with you which approach fits your content. You can find an overview of what we do under Services, or book a no-obligation intro call right away.