Storage versus presentation, the idea behind context management

Before any trick for shrinking the context, one idea has to be in place, and everything else is built on it.

The idea is a separation: what the agent stores is kept whole and never touched; what the agent shows the model is a view derived from it, and that view is what gets trimmed.

Two roles that got merged

Our agent so far uses one list for two jobs. ExecutionContext.events is both the record of everything that happened and the thing sent to the model each step. Those are different jobs, and merging them is what makes compression scary: if trimming the context means editing the record, you are destroying history to save space.

Split them and the fear goes away.

• ExecutionContext.events is the ground truth. A complete, ordered, immutable record of the run. Nothing is ever removed from it. This is your audit trail, your debugging log, your source of truth.
• LlmRequest.contents is the presentation. A view built from the events for one specific call. This is where compression happens: drop old messages, summarise, replace a file's contents with a reference. The events stay whole; only the view is trimmed.

Events are the immutable record. Contents are a trimmed view built fresh for each call.

Why this makes compression safe

The recurring worry with any compression is "what if I throw away something the agent needs?" The separation answers it.

Because the events are never altered, no compression can lose data permanently. The worst a bad compression can do is build a poor view for one call, and the next call builds a fresh view from the untouched record. A file you compacted out of the context is still in the events, ready to be pulled back in if a later step needs it.

So compression stops being a destructive edit and becomes a rendering choice. You keep the run's full history and decide what slice of it this particular call gets to see.

def build_contents(events: list[Event], budget: int) -> list[ContentItem]:
    """Derive the LLM view from the immutable record, within a token budget."""
    contents = flatten(events)           # start from the full record
    while count_tokens(contents) > budget:
        contents = compress(contents)    # trim the VIEW, never the events
    return contents

Read the comment on the loop. events goes in and is never mutated; contents is a separate list that gets trimmed until it fits. The record and the view are two objects, and only one of them shrinks.

You cannot manage what you cannot measure

The loop above stops at a token budget, which means you need to know how many tokens a view holds. That is the other half of this post, and it is easy to get vaguely wrong.

A token is a subword piece, smaller than a word and larger than a character. Models break text into these pieces, so "tokenization" is a few tokens and a rare word may be several. Estimating from character count works until the day it does not, and the day it does not is the day a call quietly exceeds the window.

Count properly, with the model's own tokenizer.

def count_tokens(contents: list[ContentItem]) -> int:
    """Count tokens the way the model will, not by guessing from length."""
    text = render(contents)
    return len(tokenizer.encode(text))    # the real count, per the model

The real count is what every compression strategy checks against. Measure the view, compare to the budget, compress if it is over, measure again. Guessing from string length is how a context manager that looked fine in testing overflows in production.

Measure the view against a budget, compress if over, measure again. The record is never counted down.

What to take from this

• Separate the record from the view. ExecutionContext.events is the immutable ground truth; LlmRequest.contents is a trimmed view derived from it for one call.
• The separation makes compression safe. Trimming the view can never lose data, because the events are untouched and the next call rebuilds the view from them.
• Measure the view with the model's real tokenizer, not a character-count guess, because a bad estimate overflows the window in production.

With the record protected and a real token count in hand, compression is now just a set of rules for building a smaller view. The next post lays out four of them, and the order to apply them.