Batch compression

Use compress_batch to compress a list of contexts in one HTTP call. It fits any batch that shares one compression model - typically RAG re-ranking against a single user question, or independent documents each with their own query. Batches are capped at 100 inputs per request. Wire-level request and response schemas: batch endpoint reference.

Python also has compress_batch_async with the same signature. TypeScript's compressBatch is already async and returns a Promise.

Why batch beats N parallel calls

• One HTTP round trip instead of N. TLS handshake, request parsing, and response framing happen once.
• One billing transaction in usage logs instead of N.
• Friendlier rate limits. A batch of 10 counts as one request against your per-second limit, not 10.
• Simpler retries. One retry decision for the whole batch instead of partial state across N in-flight requests.

The tradeoff: a batch is atomic. If any single context fails validation, the whole request fails with 422 Unprocessable. For mixed or untrusted inputs, sanitize first, or use parallel single calls so one bad item doesn't sink the rest.

Shape 1: contexts + queries

The most common batch shape is RAG re-ranking: several retrieved chunks, one user question, each chunk filtered against that question. Pass queries as a single string to apply it to every context. Pass a list the same length as contexts to give each context its own question.

Shape 2: inputs=[{context, query}, ...]

The SDK also accepts the wire-level form: an inputs list of {context, query} pairs. Use contexts + queries for re-ranking shapes (many contexts, one query, or aligned lists). Use inputs= when each item has its own query, or your data already comes as pairs. The two forms are mutually exclusive - pass one or the other.

Response shape

The response is one envelope with a results array and aggregate totals: count, plus total_original_tokens, total_compressed_tokens, and total_tokens_saved summed across the batch. average_compression_ratio is the mean of per-item ratios, not a sum. Each results entry has the same fields as a single compress call, except target_compression_ratio. That one is request-level and applies to every item.

Full response schema: batch endpoint reference; batch size and cost limits: pricing estimate endpoint.

Batch-level knobs

Most knobs on single compress also work on compress_batch and apply to every item: target_compression_ratio, coarse, plus the latte_v2 adaptive trio dynamic, dynamic_min_ratio, dynamic_max_ratio. heuristic_chunking and disable_placeholders are not accepted on the batch endpoint. Semantics match the single-call form; see POST /compress/question-specific/. To vary a knob per item, or to use the single-call-only knobs, use parallel single calls.

Scaling beyond a single batch

If you have more contexts than fit in one batch, send several batch requests in parallel instead of one giant one. A few dozen contexts per batch, with 4-8 batches in flight, usually works well. Each request stays cheap to retry while you still amortize HTTP overhead.

When to use batch vs parallel single calls