A Vyth's Note about: How to Select a Chunking Strategy for a RAG System

Last month a friend showed me a RAG prototype that kept returning half-answers. The model was fine. The embeddings were fine. The bug was hiding one layer earlier — in how she'd sliced her documents. That's the thing about chunking: it sounds like a housekeeping step, but it quietly decides how much of your retrieval you get back. I want to walk through the four strategies I keep reaching for, and the moments each one actually earns its keep.

Split by fixed-size chunk

Why do so many teams still start with the crudest chunking strategy imaginable — equal-length pieces of, say, five hundred tokens each? It's fast, it's trivial to implement, and that's exactly why most teams start here. But it splits blindly. A sentence can get sliced in half, and the context at the boundary just evaporates. The usual patch is to let chunks overlap a little, maybe ten percent, so the tail of one piece carries into the head of the next. It rescues some of the context, but you pay for it in storage.

Split by structure

Most teams skip past this strategy on the way to something fancier, and that's usually a mistake — because respecting the shape of the document does more work than any embedding trick. You split by paragraph first; if a paragraph is still too long, you keep going by sentence. Because you're following the natural boundaries the author already drew, the resulting chunks feel much more coherent than the fixed-size version. For most prose, this is a reasonable starting point — and often it's all you need.

Split by meaning

Here's the strange part about the highest-quality chunking strategy on this list: I still don't reach for it first, and by the end of this section you'll see why. Instead of leaning on punctuation, it groups sentences by what they're actually about. The system computes an embedding for each sentence, then starts a new chunk whenever the topic shifts sharply. What you get back are chunks that each stay centered on a single idea. The catch is cost: embedding every sentence in a large document store is slow and expensive, so this one isn't always worth it. I reach for it when retrieval quality really matters and the corpus is small enough to justify the compute.

Split by file type

Finally, some documents come with structure you'd be silly not to use. A markdown file splits cleanly by headers. Source code splits by class or function, not by arbitrary line count. In other words, the best strategy often depends on what you're actually processing — and the format itself will tell you where the seams are, if you listen.

Recommendation

Whichever path you take, a few rules hold up across all of them. First, attach metadata to every chunk — the document name, the section heading, whatever helps you filter and trace later. You'll thank yourself the first time you have to debug a weird retrieval. Second, don't tune chunk size blindly. Measure it against your own data. A size that works beautifully for legal documents can be terrible for conversational transcripts. Third, remember chunking isn't a one-time decision. As your corpus grows, revisit the strategy and adjust.

So, no single best strategy exists. Fixed-size wins on speed, semantic splitting wins on quality, and most real projects live somewhere in between. Start simple, measure carefully, and only add complexity when the data actually asks for it.