Chunking
Divide large documents into smaller, meaningful sections.
Explanation
Chunking breaks long texts into smaller pieces (chunks) so that they can be saved as embeddings and found effectively. The chunk size affects the quality of the search.
How it works
For example, a 100-page manual is divided into paragraphs or thematic blocks of 200-500 tokens. Each chunk is saved as its own embedding.
Example
A customer service manual is broken down into individual FAQ entries so that when a question arises, the system finds exactly the relevant section.
Why it matters
Good chunking determines the quality of RAG systems. Chunks that are too large dilute the search, and chunks that are too small lose the context.
Chunking meaning: what is chunking in AI?
In AI, chunking means splitting long documents into smaller, self-contained text segments called chunks. Each chunk is converted into an embedding and stored in a vector database. When a user asks a question, the system retrieves the most relevant chunks and passes only those to the language model.
Chunking matters because retrieval can only find what has been cut sensibly. If an answer is split across two chunks, or buried in a chunk full of unrelated content, the model never sees it. Outside AI, the word has other meanings: in psychology, chunking is a memory technique that groups information into meaningful units, and in classic natural language processing it refers to identifying short phrases within a sentence.
Chunking strategies and chunk size
The simplest method is fixed-size chunking with overlap: the text is cut every 500 tokens or so, and neighboring chunks share a few sentences so context at the boundaries is not lost. Structure-based chunking splits along headings, paragraphs, lists and tables, keeping together what belongs together. Semantic chunking uses embeddings to detect where the topic changes and places the boundary there. Another technique prepends a short context sentence to each chunk that situates it within the full document, which Anthropic described in 2024 as contextual retrieval.
There is no universal chunk size, and common values range from 200 to 1,000 tokens. Small chunks match precisely but lose context. Large chunks keep context but dilute similarity search. The right setting depends on your documents and typical questions, so measure it with a test set of real queries.
Frequently asked questions
What is chunking in RAG?
In RAG, chunking is the preprocessing step that splits source documents into smaller passages so they can be embedded, searched and passed to the language model as context.
Do I still need chunking with large context windows?
Usually yes. Even if a model can read very long inputs, sending entire document collections with every request is slower, more expensive and often less accurate than retrieving only the relevant passages.
What is the best chunk size?
There is no single best size. Many systems start somewhere between 300 and 800 tokens with some overlap and then tune based on evaluation results for their own documents.