LiteLLM

Drop the first-party Compresr guardrail into the LiteLLM proxy to auto-compress tool outputs across every provider.

LiteLLM abstracts 100+ providers behind one completion() interface. The Compresr Python SDK ships a first-party guardrail under compresr.integrations.litellm that plugs into the LiteLLM proxy server as a pre_call hook, compressing bulky tool/function messages query-aware before the request is forwarded upstream. No application changes are required: any client that calls the proxy gets the savings transparently, and the same config works against OpenAI, Anthropic, Bedrock, Vertex, Ollama, or anything else LiteLLM supports.

1. Install and register

In the environment where the LiteLLM proxy runs:

bash

LiteLLM's proxy walks litellm/proxy/guardrails/guardrail_hooks/<name>/ to discover guardrails, so until the upstream PR registering compresr directly merges, it needs to be registered explicitly. The package ships two console scripts that handle this — pick one:

bash

COMPRESR_AUTO_INSTALL_SHIM=1 also installs the shim automatically on first import compresr.integrations.litellm, if you'd rather not run a separate step. Once the LiteLLM PR merges upstream, both paths become no-ops.

2. Enable the guardrail in `config.yaml`

Add a guardrails: entry to your LiteLLM proxy config and start the proxy as usual (litellm --config config.yaml). Calls that hit a route with default_on: true (or that explicitly list compresr in their guardrails:) run through the guardrail before going upstream.

yaml

All optional keys live under litellm_params (defaults shown):

Key	Default	What it does
`api_key`	env `COMPRESR_API_KEY`	Compresr API key (`cmp_...`). Missing both → guardrail fails to load.
`api_base`	env `COMPRESR_BASE_URL` or `https://api.compresr.ai`	Override for self-hosted/on-prem.
`timeout`	env `COMPRESR_TIMEOUT` or `10.0` (seconds)	HTTP timeout for the compress call. Invalid env values warn-log and fall back to default.
`compression_model_name`	`"latte_v1"`	Query-specific compression. The guardrail's own default is `latte_v1`; set `latte_v2` explicitly (as the config above does) to get the recommended model.
`target_compression_ratio`	`0.5`	`0 < r ≤ 1` removal strength; `r > 1` Nx factor. See Models.
`target_ratio_by_role`	unset	Per-role ratio overrides, e.g. `{"system": 0.3, "tool": 0.6}`. Roles not listed fall back to `target_compression_ratio`.
`coarse`	`true`	Paragraph-level (faster); set `false` for token-level.
`min_chars_to_compress`	`500`	Skip messages shorter than this (avoids latency on trivial messages).
`compress_tool_outputs`	`true`	Compress `tool` / `function` result messages.
`compress_system`	`false`	Opt-in: compress the system prompt.
`compress_history`	`false`	Opt-in: compress prior (non-last) user messages.
`compress_last_user`	`false`	Opt-in: also rewrite the last user message (the query sent to Compresr is always the verbatim original).
`fail_closed`	`false`	On a Compresr availability error (timeout/connection/5xx), forward the original uncompressed request. Set `true` to raise instead. Validation and auth errors are always raised.
`cache_ttl`	`300` (seconds)	How long to cache a compression result in LiteLLM's `DualCache`, keyed by `(content, query, model, ratio, coarse)`. Saves a round-trip when the same tool output repeats in an agent loop. Instance-level only — not overridable per request.

3. What gets compressed by default

The defaults are deliberately conservative — only messages that are usually pure data:

Compressed: tool / function result messages (search hits, RAG dumps, API responses).
Skipped: system, assistant (model reasoning, never compressed regardless of flags), prior user turns, the last user message. Flip the matching compress_* flag to opt in.
Multimodal messages: for list-of-parts content, the text parts are extracted, compressed, and written back into the first text part; non-text parts (images, audio, files) pass through untouched.

Per-target intent query

For tool / function result messages, the query sent to Compresr is not the last user message; it's the originating tool call's name + arguments, rendered as a single string:

text

The guardrail walks back from the target message through assistant turns, matches by tool_call_id (or legacy function_call.name), and renders the matched call, falling back to the most recent user message if no assistant turn matches. This produces a much more specific compression query than "what the user asked overall." For system / prior-user / last-user compression, the query is the verbatim last user message.

4. Per-request overrides

Clients can override any optional config field for a single request by setting metadata.guardrail_config:

python

Override-able keys: compression_model_name, target_compression_ratio, target_ratio_by_role, coarse, min_chars_to_compress, compress_tool_outputs, compress_system, compress_history, compress_last_user, and fail_closed. api_key, api_base, timeout, and cache_ttl are instance-only.

5. Observability

After a successful compression pass, the guardrail stashes aggregate stats under data["metadata"]["compresr_stats"] — use these for savings telemetry:

python

It also adds itself to LiteLLM's standard applied-guardrails header — use it as a quick smoke test that the guardrail fired:

text

If Compresr is unavailable and fail_closed: false (the default), the guardrail forwards the original uncompressed request and marks the header compresr:fail_open, so you can tell "didn't fire" apart from "fired but failed open".

Releasing resources

If you construct CompresrGuardrail directly (outside the proxy's own lifecycle), call await guardrail.aclose() to release its underlying HTTP client. The proxy itself doesn't currently call this on shutdown.

6. Manual client-side pattern

If you aren't running the proxy and just want to compress a payload before calling litellm.completion() directly, use the SDK like you would in any other framework:

python

Switching providers is the usual one-line LiteLLM change to model=; the Compresr step is unchanged.

Hermes: the same query-aware compression built into the Hermes Agent.
LLM provider recipes: the manual pattern called directly against OpenAI, Anthropic, Gemini, or a local Ollama.
Models: latte_v2 parameter semantics.