Some links here are affiliate links — if you buy through them we may earn a commission at no extra cost to you. It never changes what we recommend. Full disclosure.
Disclosure: This article contains affiliate links. If you sign up through them, we may earn a commission at no extra cost to you — thanks for supporting the site.
Retrieval-augmented generation is one of the easiest places to quietly waste an enormous number of tokens, because the waste is buried inside "context" rather than an obviously bloated prompt. Every retrieved chunk you stuff into a prompt is billed, whether or not the model actually needed it.
Why This Matters
A typical RAG pipeline retrieves several chunks of source material and pastes them into the prompt alongside the user's question. The instinct is often "retrieve generously, let the model figure out what's relevant" — but that instinct is expensive, because the model pays (in tokens, and you pay in dollars) to process every chunk you hand it, relevant or not.
Common sources of waste in RAG specifically:
- Retrieving more chunks than the question actually needs "just in case"
- Overlapping or duplicate chunks from poorly deduplicated retrieval
- Sending full documents when only a paragraph is relevant
- Re-retrieving and re-sending the same context on every turn of a multi-turn conversation
A Simple Framework
- Tune your retrieval count — test whether 3 chunks perform as well as 8 for your actual queries
- Deduplicate retrieved chunks before they hit the prompt, especially with overlapping document splits
- Rerank before truncating — use a lightweight reranking step to keep only the most relevant chunks, cutting the rest
- Use a stable session or task identifier for multi-turn RAG conversations, so unchanged context can benefit from cache-aware routing instead of being fully reprocessed every turn
- Measure answer quality against chunk count, not just relevance scores — more context doesn't always mean better answers
Example
Before: A documentation chatbot retrieves the top 10 chunks for every question and pastes all of them into the prompt, regardless of relevance score, on every single turn of a conversation — including chunks that scored barely above the retrieval threshold.
After: The same chatbot retrieves the top 10, reranks them, and keeps only the top 3 that clear a meaningful relevance bar. A stable session identifier lets repeated context across conversation turns benefit from cache-aware routing instead of being billed as brand-new tokens each time.
> Tip: Log your relevance scores alongside your final answers for a sample of real queries. If the model performs just as well ignoring your 6th, 7th, and 8th retrieved chunks, that's tokens you can stop sending immediately.
Common Mistakes
- Retrieving generously "just in case" without testing whether it actually improves answers
- Skipping deduplication, sending near-identical chunks from overlapping document splits
- Resending the same static context every turn instead of leveraging session-based caching
- Optimizing retrieval relevance without ever measuring the token cost of the retrieval count chosen
DigitalOcean's Inference features documentation covers cache-aware routing and session affinity, both directly relevant to multi-turn RAG cost control.
> (ad) GPU bills piling up for AI features you barely use? DigitalOcean Serverless Inference is pay-per-token, no idle infrastructure, no minimums. Ship the feature, not the server bill.
RAG pipelines don't get more expensive because the model is doing more work — they get expensive because we hand the model more context than it needed. Tightening retrieval is almost always cheaper than switching to a smaller model.