The real cost of RAG (spoiler: it's not the tokens)

Most teams price their RAG stack by looking at LLM token costs. That's the cheap part. Embedding a document with OpenAI's text-embedding-3-small costs $0.02 per million tokens — roughly 100x cheaper than a generative model call on the same volume. If your monthly bill is dominated by embeddings, something else is wrong.

The real cost of RAG lives in four other places: initial embedding, ongoing storage, vector database hosting, and — the one almost nobody checks — re-embedding documents that never actually changed.

1. Your true monthly cost, broken down by line

Here's what a mid-size deployment actually looks like — 10 million document chunks, 1,536-dimension embeddings, moderate query volume:

That last line is the one that swings 10x depending on architecture — and it's the one most cost estimates leave out entirely. Storage is also worth watching: one pricing analysis found that for large corpora, the monthly storage bill often overtakes the one-time embedding cost within a few months, since embedding is a single charge and storage is forever.

2. Where the money quietly leaks

Three leaks show up over and over in production RAG bills:

Storage compounds, embedding doesn't. You pay for embedding once. You pay for storage every month, for as long as the vectors exist. At 10M vectors, storage alone typically runs $30–135/month depending on provider — and it only goes up as the corpus grows, never down, unless someone actively prunes it.

Vendor pricing pages exclude a third of the real bill. Vector database pricing calculators commonly leave out data import, reranking, and — critically — reindexing overhead. Teams that budget from the pricing page alone tend to land at 2.5–4x their estimate once the system is actually in production, according to one infrastructure audit of AI teams' vector DB bills.

The embedding model you pick early gets expensive to leave. Switching from a legacy model like ada-002 ($0.10/M tokens) to text-embedding-3-small ($0.02/M) pays for itself in a few months at volume — but only if you actually do it. Most teams never revisit the choice, and pay the 5x premium indefinitely.

Retrieval iterations never show up on a pricing page. This one's easy to miss because it isn't a separate line item — it inflates your LLM bill instead. When retrieval quality is poor, the pipeline doesn't fail cleanly, it retries: a reformulated query, another fetch, another round after a low-confidence result. Each retry multiplies the token cost of that one query, invisibly. A related breakdown of search-API costs makes the general case well: the right unit to benchmark retrieval on isn't price per call, it's cost per resolved query — total tokens burned across searching, fetching, and retrying to get from question to a verified answer. Applied to RAG specifically, a retrieval layer that looks cheap per query but forces extra rounds is the expensive one, and that gap is invisible on any vendor's pricing calculator. The fix is the same discipline as section 3: measure at the level that matters. Run real queries end to end, count every token from question to accepted answer, divide by query count — that number is your actual retrieval cost, not the advertised price per call.

3. How much you pay to re-process data that never changed

This is the "red line" — the one that's easy to miss because it doesn't show up as a separate bill. It's baked into your embedding API costs, disguised as normal usage.

The default architecture most RAG tutorials teach is: on any update, delete everything, reload every document, re-chunk everything, re-embed everything. If you have 50,000 documents and re-index nightly on that pattern, you pay for 50,000 embedding calls every night — regardless of whether 1 document changed or 10,000 did.

Compare that to change-detection: hash each document's content, and only re-embed when the hash changes. If 1% of a 50,000-document corpus actually changes per day, that's 500 embedding calls instead of 50,000 — roughly a 100x reduction in daily embedding spend, for identical retrieval quality. Nothing about the answers gets worse; you've just stopped paying to re-learn what you already knew.

This compounds at the architecture level too. Teams that go through a chunking-strategy change or an embedding-model upgrade typically trigger 1–3 full re-indexes in their first six months — each one costing the full corpus's embedding price again. One production post-mortem put sprint-to-sprint re-indexing costs at $50–600+ for a few-million-document corpus, from changes that weren't in the original budget at all.

4. The cost you can cut without touching quality

None of these require a worse model or worse retrieval — they're architecture and configuration choices:

• Hash-based change detection. Only re-embed documents whose content hash actually changed. This is the single highest-leverage fix, and it's the one from section 3.
• Batch API for indexing. Both OpenAI and Voyage offer 33–50% discounts for asynchronous batch embedding — the right default for any initial load or scheduled re-index where overnight latency is fine.
• Right-size the embedding model. text-embedding-3-large costs 6.5x more than text-embedding-3-small for roughly 2–3 MTEB points of retrieval quality — a premium that rarely pays for itself outside domain-specific or non-English content.
• Matryoshka dimension truncation. Modern embedding models let you request fewer output dimensions (e.g., 512 instead of 1,536) with a small, predictable quality cost — cutting vector storage proportionally, since storage scales directly with dimension count.
• Quantization on the vector DB side. Binary quantization can compress vector storage by roughly 32x on the same hardware, which is often the difference between needing a bigger (pricier) cluster and not.
• Match your DB pricing model to your query pattern. Consumption-based pricing (Pinecone-style) is cheaper at low, spiky query volume; capacity-based pricing (Qdrant-style) gets cheaper as query volume grows, since you're not paying per-read once you own the node.

The pattern underneath all of this

Every leak on this list traces back to the same root cause: paying for compute proportional to your total corpus size, when the real cost driver is your change rate. A corpus that's 99% stable on any given day should cost close to nothing to maintain — but most default RAG architectures charge you as if the whole thing changed every time.

The fix isn't a better model. It's checking, line by line, whether your bill actually reflects what changed — or just how much you have.

Ready to give your agents the context they need? Get started for free or talk to our team.

Further Reading

• OpenAI embedding pricing, verified monthly — embeddingcost.com
• On benchmarking retrieval by cost per resolved query rather than price per call — Linkup: Your search API isn't what's expensive, your tokens are