Building a Token-Efficient RAG Pipeline: Lessons from Production

The Problem

Our RAG pipeline was burning $400/day on embeddings and completions. After optimization, we got it down to $45/day with better results.

What Worked

1. Semantic Chunking > Fixed-Size Chunking

Instead of splitting at 512 tokens, we split at paragraph boundaries and merged small chunks. Retrieval precision jumped 23%.

2. Two-Stage Retrieval

Query → BM25 (top 50) → Reranker (top 5) → LLM

BM25 is nearly free. The reranker catches what embeddings miss.

3. Context Compression

Before sending retrieved chunks to the LLM, we compress them: - Remove boilerplate headers/footers - Deduplicate overlapping information - Summarize long chunks that are only partially relevant

4. Caching at Every Layer

• Embedding cache (Redis)
• Query cache (semantic similarity threshold)
• Response cache (exact match + TTL)

Results