chunking

RAG systems cannot feed a whole website into a model, so they break sources into chunks (often no more than a few hundred tokens each), embed every chunk separately, and store the vectors. At query time the retriever fetches the closest chunks and hands them to the model as the context it answers from.

Why chunk size and structure decide what gets retrieved

Chunk size is the main lever. Split too small and a passage loses the context it needs to make sense. Split too large and the embedding averages several topics into one blurry vector that matches nothing well. Practitioners commonly land near 400 to 512 tokens with 10 to 20 percent overlap, so ideas that cross a boundary are not cut in half.

Common strategies, roughly in order of effort:

• Fixed-size: split by token count. A simple baseline.
• Recursive: split on paragraphs, then sentences. The pragmatic default.
• Semantic: place breaks where the topic actually shifts.
• Contextual: prepend a short note situating each chunk in its document.

The failure mode is context loss. Anthropic’s example: a chunk reading The company's revenue grew by 3% over the previous quarter is useless if nothing in that chunk says which company or which quarter. Contextual chunking prepends the missing context before embedding, and Anthropic reports it cut top-20 retrieval failures by up to 49% when paired with a BM25 keyword index.