Chatbot Context Windows: How Much Conversation to Keep

Two model releases this week put million-token context windows back in the spotlight. Thinking Machines Lab says Inkling supports up to one million tokens, while Kimi describes Kimi K3 as a one-million-token model for long-horizon work.

That capacity is impressive, but a website chatbot still needs a context policy. Passing everything into every request makes stale turns, conflicting sources, private details, and repeated tool output compete with the customer’s current question.

Start with five buckets and a limit for each:

This is the working asset: a chatbot context budget. The model’s advertised maximum is only the ceiling. Your operating budget should be smaller, observable, and tested against the conversations customers actually have.

Treat Context as a Working Set

A context window is the material available to the model for one response. It can include system instructions, developer rules, the current message, previous turns, retrieved source passages, account data, and tool results. Every item competes for attention inside the same request.

Context is different from memory. Memory decides what can persist or be found later. Context decides what the model can see now. A business may store a year of authorized conversation history while showing the model only the current issue, a short summary of relevant prior decisions, and the last few turns.

That distinction also prevents a common design error: treating a larger window as permission to stop curating. Long context can make a difficult request possible, but it does not make every old message equally relevant. A resolved shipping question from March should not distract from today’s billing dispute. An outdated policy page should not remain beside its replacement just because both fit.

The chatbot memory guide begins one layer earlier with identity, consent, and retention. Once those rules decide what may be remembered, a context policy decides what deserves to be recalled for this turn.

Set a Budget Before You Need the Maximum

Do not start with the provider limit and fill backward. Start with the response you need to produce, reserve room for it, then allocate the input around the task.

Suppose a support chatbot uses a model with a 128,000-token window. The team sets a normal operating budget of 24,000 input tokens and reserves 4,000 tokens for the answer and tool planning. A typical request might look like this:

The total request uses 28,000 tokens including the reserved output, leaving most of the model’s maximum untouched. That headroom is deliberate. It reduces truncation risk, keeps latency and cost more predictable, and allows an exceptional case to expand without turning the maximum into the default.

Track actual usage by bucket rather than logging only a total. If retrieved sources consume 70% of the request, fix retrieval. If conversation history grows every turn, add compaction. If tool output dominates, store the raw response outside the prompt and pass back a structured result.

Preserve Decisions, Not Every Sentence

A useful conversation compactor creates a state record instead of an elegant recap. It should retain facts that change the next answer and discard language that merely led to them.

For a support conversation, keep:

• Active intent: what the customer is trying to accomplish now.
• Verified facts: order ID, selected plan, device, dates, or other confirmed fields.
• Corrections: “annual plan, not monthly” matters more than the earlier mistaken value.
• Decisions and promises: what the chatbot or a human already said would happen.
• Open questions: information still needed before an answer or action.
• Source and tool references: identifiers needed to reproduce the result.

Do not summarize unverified model claims as facts. If the bot guessed that a purchase was refundable, the compacted record must say “refund eligibility not verified,” not “customer qualifies for refund.” Otherwise one weak answer becomes trusted context for every later turn.

Keep the latest raw turns beside the structured state. Summaries lose tone, exact wording, and sometimes negation. The last four to eight turns usually provide enough local texture for the model to interpret “yes, that one” or “I already tried it,” while the state record carries older facts.

Version the summary whenever it changes. A context trace should show which messages were compressed, what state replaced them, and which rule performed the change. That makes a failure diagnosable instead of leaving reviewers to wonder what the model saw.

Retrieve Evidence for the Current Question

Training a chatbot on a large website or document library does not mean inserting that library into every prompt. Retrieval should select the smallest set of authoritative passages that can support the current answer.

Use three gates:

Relevance: does the passage answer the active question, or does it merely share keywords? A pricing page mentioning “security” is not necessarily evidence for an SSO question.

Authority: which source wins when several passages conflict? Policy pages, controlled Q&A pairs, and current product documentation should outrank an old announcement or a user-generated comment.

Freshness: is the passage still valid? Store update dates and invalidate replaced content. More context makes a contradiction larger; it does not resolve it.

The source-pipeline guide covers extraction and ranking for PDFs, pages, and tables. At request time, the final discipline is subtraction: deduplicate overlapping chunks, drop weak matches, and stop retrieving once the evidence answers the question.

When no source clears the threshold, pass that absence into the prompt. “No approved source found for enterprise data residency” gives the chatbot a clear reason to ask, qualify, or escalate. An empty evidence field that looks like success invites a plausible invention.

A Worked Long-Conversation Repair

Consider a customer who starts with pricing, switches to security, and returns to pricing 35 messages later. A naive chatbot sends the full transcript plus five complete web pages. Near the end, it answers from the first pricing discussion and misses the customer’s correction.

The useful state after compaction is much smaller:

{
  "active_intent": "compare Standard and Enterprise for SSO",
  "verified_customer_facts": {
    "team_size": 42,
    "billing_preference": "annual"
  },
  "corrections": [
    "Customer needs SSO, not social login",
    "Compare annual pricing, not monthly pricing"
  ],
  "open_questions": [
    "Is SSO available on Standard?",
    "Does Enterprise require a sales quote?"
  ],
  "claims_not_yet_verified": [
    "A previous bot turn suggested SSO might be included on Standard"
  ]
}

The retrieval layer then supplies the current plan comparison and SSO policy. The model sees the state, the last six turns, and those two source passages—not the whole pricing page, 35 old messages, and unrelated security documentation.

The repaired answer can be short and traceable:

SSO is not included on the Standard plan according to the current
plan comparison. Enterprise includes SSO and requires a sales quote.

You said you are comparing annual billing for a 42-person team. I can
summarize the remaining plan differences or help you contact sales.

The correction survives, the unsupported earlier claim stays labeled, and each sentence maps to either verified conversation state or retrieved evidence. That is better context management than merely fitting the transcript into a larger window.

Give Tool Results an Expiration Rule

Tool output can quietly become the largest and stalest part of a chatbot request. A product search may return 50 records. An order API may include hundreds of fields. A failed action may be retried with the same error payload attached each time.

Convert raw output into a narrow result contract:

• tool name and execution ID;
• authorized subject or account scope;
• status and timestamp;
• fields needed for the next decision;
• error class and whether a retry is safe;
• link to the full protected trace outside the prompt.

Then expire it. Inventory, delivery estimates, account balances, and queue positions can change during a conversation. A result that was correct ten minutes ago may need a fresh lookup before the chatbot makes a promise.

For actions, preserve the idempotency key or downstream record ID so trimming the raw payload does not encourage a duplicate execution. The audit-trail guide shows the wider trace needed to reconstruct an action without stuffing that whole trace into every following prompt.

Test the Context Policy Under Pressure

A short FAQ test will not reveal long-context defects. Build conversations that force information to compete.

Record the prompt composition for each test: bucket sizes, selected source IDs, summary version, discarded-message count, and final token total. Score factual accuracy, correction retention, source support, latency, and whether private or irrelevant content appeared.

Run the suite when you change models, retrieval settings, chunking, prompts, summarization logic, or tools. A model with a larger advertised window may need a different budget, but it should not receive a free pass on the same behavioral tests.

Keep the Window Deliberately Smaller Than the Conversation

Long context expands what a chatbot can do. It does not choose what the chatbot should remember, what evidence deserves authority, or when a tool result has gone stale. Those are application decisions, and they need explicit limits.

A good context policy preserves the active intent, verified corrections, current evidence, and the minimum state required for the next safe action. Everything else remains retrievable, auditable, or intentionally forgotten outside the model’s immediate working set.

In Agentkit, source training, prioritized Q&A pairs, conversation logs, and analytics provide the material for reviewing answers without assuming that every stored item belongs in every response.

Build your chatbot for free →

No credit card required.