Skip to content
Compresr docs

Guides

RAG integration

Add Compresr between retrieval and the LLM call to shrink retrieved context against the user's question.

Add Compresr between retrieval and your LLM call to shrink retrieved chunks against the user's question. A typical RAG pipeline retrieves top-k chunks, joins them, and puts them in the prompt. Compresr adds one step in the middle: it scores the chunks against the question and passes only the parts that answer it to the LLM. Your vector DB, embedding model, and LLM stay the same.

Pipeline shape

text

The query you pass to Compresr is the string the user typed. The context is the retrieved chunks, joined. With batch, pass them as a list instead.

With LangChain: CompresrExtractor

Use CompresrExtractor, the SDK's first-party BaseDocumentCompressor. Drop it into ContextualCompressionRetriever and retrieved documents are compressed against the retrieval query in one batched Compresr call.

python

For LangChain LCEL / RunnableSequence setups, CompresrExtractor also implements acompress_documents. The LangChain integration page has the full option table (drop_below_min, batching behavior, metadata tagging) and the agent middlewares for tool-output, history, and outbound-prompt compression.

With LlamaIndex: CompresrNodePostprocessor

Use CompresrNodePostprocessor, a BaseNodePostprocessor. Pass it to as_query_engine or any RetrieverQueryEngine. Retrieved nodes are compressed against the query before synthesis sees them. Originals in the index stay untouched.

python

The LlamaIndex integration page has the full option table (including target_token for absolute per-node token budgets), plus tool wrapping and memory-block compression.

Direct vector DB (no framework)

You can call a vector DB directly (pgvector, Pinecone, Qdrant, Chroma, Weaviate) and keep the same shape: retrieve, compress, pass to LLM. Use compress_batch / compressBatch to filter each chunk against the same question in one HTTP round trip. That's cheaper than N parallel compress() calls.

python

For hybrid search or multi-hop RAG where each chunk has its own question, pass queries as a list the same length as contexts, or use the inputs=[{"context": ..., "query": ...}, ...] pair form. contexts / queries is SDK sugar; the wire format is always inputs, which is what the cURL tabs send. Input shapes, atomicity, and the async twin are covered in the batch guide.

Tips

  • Use the framework-native integration when you can. CompresrExtractor (LangChain) and CompresrNodePostprocessor (LlamaIndex) handle batching, node cloning, min_tokens partitioning, and error policy for you. Manual calls work, but you rebuild that wiring.
  • Pick a ratio that matches your token budget. Lighter ratios (around 0.3-0.5) keep enough surrounding context for citation-style or extractive Q&A. Heavier settings (0.7+, or Nx like 4) work for summarization-style answers. Start at 0.5, measure on a held-out set, then tune. Ratio semantics and bounds: Models reference.
  • Put the compressed text in the system message. It's reference material, not a turn in the conversation. Keep the user's question in the user message. See the LLM provider recipes for the exact slot per provider.

When NOT to compress

  • Tiny contexts. Under ~500 tokens, the API call overhead isn't worth the savings. Raise min_tokens to skip them automatically.
  • Tightly structured retrieval results like JSON, tool outputs, and schemas. Compresr is tuned for prose. For structured payloads, turn on heuristic_chunking and disable_placeholders to reduce field-loss risk, or route tool outputs through CompresrToolMiddleware instead.

Variable chunk sizes: latte_v2 + dynamic ratios

If your chunk sizes vary a lot, say short FAQ snippets next to long PDF pages, one target_compression_ratio over- or under-compresses at the extremes. Set compression_model_name="latte_v2" and pass dynamic=True with dynamic_min_ratio / dynamic_max_ratio. The model then picks a per-chunk ratio inside that window based on input size. latte_v2-only; see Models reference.

Related