context-compressor

Shrink LLM context by 60–80% — and prove nothing important was lost.

Most of your context window is repeats. Agents re-read the same files, users restate the same rules, docs duplicate the same definitions. context-compressor removes that bloat with a deterministic, extractive engine — then validates that every token it kept is verbatim from your input and every constraint, decision, and preference survived.

_{Real output on the demo corpus. A reduction only counts if validation passes.}

TL;DR — Point it at a chat log, spec, or source file. It removes the repeats and the filler, keeps every constraint, decision and preference, and prints a verdict proving nothing important was dropped. No model, no API, no fabrication.

git clone https://github.com/ingridtoulotte/context-compressor
cd context-compressor
python benchmarks/gen_corpus.py
python -m ctxcomp benchmarks/corpus/conversation.jsonl

Prefer a smaller, human-readable example? python -m ctxcomp examples/conversation.jsonl (a realistic agent session — 66% smaller, validated).

  context-compressor  ·  mode: conversation
  ──────────────────────────────────────────────────────────
  before         2.0K tokens
  after          248 tokens

  ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  88% smaller

  effective context  8.05x  (same window now holds 2.0K of raw context)

  ──────────────────────────────────────────────────────────
  removed
    exact duplicates         30  lossless
    near duplicates           0  labeled
    low-value segments        0  by importance

  kept   12 of 42 segments
  ──────────────────────────────────────────────────────────
  validation  ✓ PASS
    kept text is verbatim  yes
    references resolve     yes
    critical preserved     12/12 kept
  ──────────────────────────────────────────────────────────

Context windows are getting bigger, but the bottleneck moved, it didn't vanish:

• You pay for every token, every call. A 200K-token history re-sent on each turn is 200K tokens billed each turn — most of it redundant.
• Naive summarization is dangerous. "Make it shorter" with an LLM drops the one constraint that mattered, paraphrases a decision into something subtly wrong, and can't be reproduced or audited.
• Different content needs different handling. A chat log, a spec, and a source file bloat in different ways and must be compressed differently.

context-compressor attacks the measurable waste — repetition and low-value filler — with a method you can inspect and trust, instead of a black box.

Semantic compression with reviewable, provable preservation of meaning.

It is extractive, not generative. It selects, references, and restructures your existing text — it never writes new text. That single constraint buys three things a summarizer can't give you:

1. No fabrication — every kept token is verbatim. Verified, not promised.
2. Determinism — same input, same flags, byte-identical output. No model, no randomness in the core.
3. Auditability — every segment's keep/drop decision and reason is inspectable.

• 🔻 60–80% smaller context on realistic, repetitive inputs
• 🔒 Lossless-first — provable dedup does the heavy lifting; lossy steps are labeled
• 🧠 Preserves what matters — constraints, decisions, preferences are never dropped
• 🧾 Reviewable output — structured blocks + a full audit log
• 💬📄💻 Content-aware — conversation, document, code, and project-memory modes
• 🖥️ Local-first, zero runtime deps — your context never leaves your machine
• 📊 Benchmarked & reproducible — honest numbers, seeded corpus, validation gate

1. Segment — split into meaningful units (turns, sections, paragraphs, code defs, tool outputs).
2. Score — rank each unit by importance and tag constraints / decisions / preferences / open questions.
3. Redundancy — remove exact duplicates losslessly (reference the first copy); detect near-duplicates.
4. Compress — drop only the lowest-importance, non-critical, non-canonical units to hit the target.
5. Validate — prove kept text is verbatim, references resolve, and all criticals survived.
6. Format — emit clean, reviewable Markdown / JSON / prompt / audit.

Full design notes: docs/ARCHITECTURE.md.

# one-liner, no clone (installs the `ctxcomp` command)
pipx install "git+https://github.com/ingridtoulotte/context-compressor"

# or from source
git clone https://github.com/ingridtoulotte/context-compressor
cd context-compressor
pip install -e .

# exact token counts (optional — heuristic is used otherwise)
pip install -e ".[exact]"

Requires Python 3.8+. The core has zero runtime dependencies — nothing to audit, nothing to break, nothing phoning home.

ctxcomp conversation.jsonl                 # auto-detect mode, print report
ctxcomp spec.md --target 0.75              # aim for 75% smaller
ctxcomp app.py --mode code                 # force a mode
ctxcomp log.jsonl --format json -o out.json
cat history.jsonl | ctxcomp -              # read from stdin
ctxcomp --selftest                         # run the built-in test suite

Or as a library:

from ctxcomp import compress, validate
from ctxcomp.formats import to_prompt

result = compress(open("history.jsonl").read(), mode="conversation", target=0.7)
report = validate(result)
assert report.ok                 # verbatim, refs resolve, criticals kept
prompt_ready = to_prompt(result) # feed this to your model instead of the raw log
print(result.ratio, result.effective_multiplier)

A 60-section spec that repeats one definition and one example block per section:

What it did: deduped 48 repeated definition/example blocks (lossless), hoisted the constraints and the single open question into dedicated blocks, kept one canonical copy of everything unique. Validation: ✓ kept text verbatim, all 48 references resolve, criticals preserved.

Honest numbers from the seeded demo corpus (exact tiktoken counts). A reduction only counts if validation passes.

Reproduce:

python benchmarks/gen_corpus.py
python benchmarks/run_benchmark.py --target 0.65 --md

Methodology, metrics, and caveats: docs/BENCHMARKS.md. Highly unique prose compresses less; repetitive agent logs and multi-section docs compress more. We don't fake numbers and we don't hide the caveats.

Every run is checked by a validation layer and the verdict is printed:

• Verbatim subset — every kept token exists, unchanged, in your input. The engine cannot fabricate, because it cannot generate.
• Reference integrity — every "this is a duplicate of #N" pointer resolves to a segment that is actually kept. Nothing is orphaned.
• Critical preservation — 100% of detected constraints, decisions, and preferences are retained (verbatim or via a byte-identical canonical).
• Ambiguity flags — anything dropped to hit your target that contained factual cues (paths, numbers, code spans, URLs) is flagged for review, never silently discarded.

Want the receipts? ctxcomp file --format audit prints every segment with its keep/drop decision, importance score, and reason.

context-compressor isn't a replacement for RAG or memory — it's the pre-processing layer that makes whatever you send next smaller and safer.

Is it really lossless? The dedup and whitespace handling are provably lossless. Importance trimming and near-duplicate removal are near-lossless / lossy and are always labeled as such in the output and audit log. We never call a compressed artifact identical when it isn't — that's why the tool is "lossless-first", not "magically lossless".

What does "semantic compression" mean here? We compress by meaning-bearing units — a repeated rule, a duplicated example, a re-read file — rather than by characters. Identical meaning carried twice is stored once and referenced; the engine reasons about constraints and decisions, not just string length.

Can it handle code? Yes. Code mode splits source into definition blocks, dedups repeated helpers, and never alters a byte inside a kept block. It produces a compact briefing, not a rewritten file.

Does it work locally / offline? Entirely. Zero runtime dependencies, no network calls, your context never leaves your machine. tiktoken is an optional extra purely for exact token counts.

How do I trust the result? Read the validation verdict (printed every run) and the --format audit log. Every decision is deterministic and inspectable.

Will it replace my RAG pipeline? No — it complements it. Use it to compress what you've already decided to send.

v0.1 is intentionally focused: a deterministic extractive engine that is correct, honest, and reproducible. Planned next:

• Optional LLM rewrite stage behind the same validation gate (true semantic rewriting where it's safe, still measured and labeled).
• Task-success benchmark (does an LLM answer identically on compressed vs. full context?), beyond today's fidelity validation.
• MinHash/LSH near-dup backend for very large corpora.
• Provider wrappers (drop-in pre-call compression for Anthropic / OpenAI).
• Handoff mode presets for moving work to a fresh session.
• Compression presets and per-project shared rules.

The core ships with zero runtime dependencies on purpose: it must be trivial to vendor into any project, safe to run on sensitive context, fast to install, and immune to supply-chain churn. The only optional extra is tiktoken, used solely to upgrade token counts from heuristic to exact.

Issues and PRs welcome. Before submitting: python -m ctxcomp --selftest and python -m pytest must pass, and any change touching the engine should keep the benchmark validation at ✅. See docs/ARCHITECTURE.md.

MIT © 2026 Ingrid Toulotte. See LICENSE.