If you blindly chop documents into fixed 500-token chunks, you are structurally guaranteeing that context gets destroyed. I was looking at retrieval failures in a domain-specific pipeline recently and it’s amazing how much of the error rate just comes down to arbitrary text splitting.
Think about what a fixed 500-token split actually does. It counts tokens and cuts. It doesn’t care that it’s slicing through the middle of a sentence, splitting a definition from the thing it defines, or stranding a table’s numbers three chunks away from the header that says what they mean. The boundary lands wherever the counter hits 500. So a fact that only makes sense as a whole gets torn in half and stored in two places, and neither half retrieves cleanly. That’s not a tuning problem. It’s baked into the method. You chose a boundary that has nothing to do with meaning, so meaning is what you lose.
Chunk on logic, not character counts
Moving to sentence-level semantic chunking adds compute overhead, but it forces the pipeline to respect logical boundaries instead of character counts. Recall immediately stabilises.
Semantic chunking splits on where ideas actually end, not where a token counter happens to land. A sentence stays whole. A paragraph that’s really one thought stays together. Yes, it costs more. You’re doing real work to find those boundaries instead of just counting to 500. But that overhead buys you chunks that mean something on their own, and the payoff shows up straight away. Recall stops bouncing around. When each chunk is a complete unit, the retriever can actually match it, because there’s a coherent idea in there to match against.
Split the search problem from the reasoning problem
A really elegant pattern to fix the rest of the gap is Parent-Document Retrieval. You run the vector search strictly over small, granular chunks, but you hand the unbroken parent document to the LLM. It cleanly decouples the search problem from the reasoning problem.
These are two different jobs and they want two different things. Search wants small, specific chunks, because a tight chunk gives a sharp embedding and a precise match. Reasoning wants context, because the model answers better when it can see the whole surrounding passage, not a lonely fragment. Most pipelines force one size to do both and lose on at least one end. Parent-Document Retrieval refuses the tradeoff. You embed and search over the small chunks, so retrieval stays precise. Then, once you know which chunk matched, you don’t hand that fragment to the model. You hand it the full parent document it came from. Search gets its precision. The model gets its context. Neither is compromised to serve the other.
Make the chunks carry their own context
If you pair this with rigorous metadata augmentation like injecting headers, summaries and source tags directly into the chunks, the retrieval filtering becomes incredibly sharp, far beyond baseline vector similarity.
Raw vector similarity is a blunt instrument. It matches on semantic closeness and nothing else, so it can’t tell you that a chunk came from the right section, the right document, or the right version. Metadata fixes that. Inject the header the chunk lived under, a short summary of what it covers, a source tag saying where it’s from, and now the chunk carries its own context. You can filter on that. Restrict to a section. Scope to a source. Cut the candidate set down before similarity even runs. The retrieval gets sharp in a way pure vector distance never manages on its own, because you’ve given it structured handles to grab instead of asking it to infer everything from the embedding.
In practice, scaling RAG performance is almost entirely bounded by the topology of your chunking and your data architecture.
That’s the real constraint. People reach for a bigger model or a better embedding when retrieval is failing, but the ceiling usually isn’t there. It’s in how the data was split and structured before any of that ran. Get the chunk boundaries right, split search from reasoning, give the chunks metadata to filter on, and the same model and the same embeddings suddenly perform. The topology of your data is doing more of the work than anything downstream of it.