VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

How Super Intelligence Works | Context Windows — How Much Information a Model Can Use at Once

Three students in school uniforms work through open books at a classroom table, with textbooks and stationery nearby and study notes on the whiteboard behind them.

A context window is the amount of tokenized information a model can use within one inference request or active generation process. It is not the same as permanent memory, training data or a database. Context is the working information made available to the model now.

A large context window changes what can fit into one task: longer conversations, more documents, richer tool descriptions and larger evidence packets. But capacity is only the first question. The system must still select relevant information, preserve source structure, prevent truncation of critical constraints and verify that the model actually uses the evidence reliably.

This article explains how context windows work in Super Intelligence from first principles. We will build token budgets, examine input and output allocation, explain truncation and chunking, distinguish context from memory, study long-context attention cost and KV-cache growth, and develop a practical context-engineering method for real tasks.

In the eduKateSG series, Super Intelligence, or SI, is our umbrella term for AI technologies. Standard terms such as context length, token budget, prompt, chat history, system instruction, retrieval chunk, truncation, KV cache and long-context evaluation remain visible so readers can connect this guide to model documentation.

Previous: 019 — Layers. Return to the How Super Intelligence Works hub for the complete series.


Context Windows at a Glance

  • A context window is measured in model tokens, not pages or human words.
  • The window can contain system instructions, user messages, prior conversation, retrieved documents, tool schemas and tool results.
  • Input and generated output compete for finite model capacity under the serving configuration.
  • Fitting information into the window does not guarantee the model will use it correctly.
  • Truncation can remove critical evidence or instructions without being obvious to the user.
  • Retrieval helps select relevant evidence instead of sending every available document.
  • KV-cache memory grows with active context during autoregressive inference.
  • Persistent memory and database state are separate mechanisms that can be retrieved into context when needed.

What Counts as Context?

Context is everything the model receives for the current inference stage. It can include instructions, conversation history, documents, examples, tool definitions and observations from earlier tool calls.

The visible user message may be only a small part of the actual context packet assembled by the application.

For debugging, inspect what the model really received rather than estimating from the chat interface alone.

Context Is Not Training Data

Training data influences model parameters through optimization before deployment. Context influences one current inference without necessarily changing those parameters.

A user can teach the model a temporary rule inside context and observe changed output immediately. That does not prove the base weights were retrained.

This distinction separates in-context adaptation from parameter learning.

Context Is Not Persistent Memory

Persistent memory stores selected information outside the active inference so it can be retrieved in later tasks.

A remembered preference may live in a database and be inserted into context when relevant. The context window then carries the retrieved memory for that one request.

Clearing the active context does not necessarily delete persistent memory, and clearing memory does not change the pretrained weights.

Context Is Not a Database

A long prompt can contain thousands of records, but it does not gain transaction semantics, exact primary-key lookup or durable update behaviour.

If current authoritative state exists in a database, query that state and provide the relevant result rather than treating the context window as a replacement data store.

The model’s working context and the application’s authoritative state should remain distinct.

The Context Budget

A context budget allocates finite token capacity among all information needed for a request.

Suppose a fictional model supports 16,000 tokens for input plus output under one serving setup. The application reserves 2,000 tokens for the answer, leaving 14,000 for instructions, history, evidence and tool descriptions.

Every added item consumes part of that budget. Context engineering is therefore a resource-allocation problem as well as a semantic-selection problem.

A Worked 16,000-Token Budget

System instructions use 1,500 tokens. Tool schemas use 1,000. Conversation history uses 3,500. The current user request uses 500. That totals 6,500 before retrieved evidence.

With 2,000 tokens reserved for output, 7,500 tokens remain available for new evidence under this simplified budget.

If retrieval returns 12,000 tokens, something must change: retrieve fewer chunks, rerank, summarize, use a larger window or shorten another component.

Input and Output Share Capacity

Some model APIs describe context limits as a combined capacity for prompt and completion; others expose separate constraints or implementation details.

The practical rule is to read the serving documentation for the actual model and reserve enough space for the desired output.

A prompt that nearly fills the window can leave insufficient room for a complete answer even though all input tokens were accepted.

System Instructions Use Tokens Too

System-level instructions, policy text and tool descriptions are part of the model input and therefore consume token capacity.

A complex agent with dozens of verbose tool schemas can spend thousands of tokens before the user supplies any task-specific evidence.

Tool design therefore affects context efficiency as well as functionality.

Conversation History Uses Tokens Too

Chat interfaces often include selected prior messages in new requests so the model can maintain continuity.

Long conversations can gradually push old instructions and evidence toward truncation or summarization boundaries.

A visible chat history on screen does not guarantee every earlier message is still inside the active model context.

Retrieved Documents Use Tokens Too

Retrieval-augmented generation selects passages and inserts them into context. The model can then attend to those passages while answering.

The retriever must balance coverage and noise. Returning twenty loosely relevant chunks can consume the budget and crowd out the few passages that matter.

Retrieval quality therefore becomes part of context quality.

Tool Results Accumulate

An agentic workflow can perform several tool calls. Search snippets, file contents, database rows and code outputs can all be appended to the context.

If every result is retained verbatim, the context can grow rapidly across steps.

A good workflow keeps the evidence needed for later decisions and discards or summarizes material that no longer contributes.

Context Windows Are Tokenizer-Specific

The same visible document can consume different numbers of tokens in different models because tokenization differs.

Code, multilingual text, URLs, numbers and OCR noise can have very different token efficiency from ordinary English prose.

Exact context budgeting should therefore use the tokenizer associated with the deployed model.

Word Count Is Not Token Count

A 5,000-word English essay can fit differently from 5,000 words of code or another language.

Rules of thumb such as “one token is roughly three quarters of a word” can be useful for rough English estimates but are not capacity guarantees.

When the task approaches the context limit, measure the actual tokenized sequence.

Truncation

When assembled context exceeds the allowed capacity, the application must remove, summarize or otherwise reduce something before model inference can proceed.

Naive truncation can remove the most important clause. If a policy ends with an exception and the system simply cuts off the final tokens, the model receives a coherent but incomplete rule.

Truncation strategy is therefore a semantic decision, not merely a technical reaction to an oversized prompt.

A Worked Truncation Failure

Policy text: “Normal camera loans are three school days. Approved field-project loans may be five school days with teacher authorisation.”

The context budget cuts off after “Approved field-project loans may be”. The model answers that all loans are limited to three days.

The first unstable point is context preparation. The model cannot apply an exception that never arrived.

Left Truncation Versus Right Truncation

Some applications drop the oldest tokens; others drop the newest; some preserve system instructions while trimming conversation history.

Each strategy protects one part of the context while risking another. Dropping old history can remove a standing user constraint; dropping the end of a document can remove conclusions or exceptions.

Good context management defines priority rather than relying on one universal direction of truncation.

Priority-Aware Context Assembly

An application can classify context into protected instructions, current request, authoritative evidence, useful history and optional background.

When the budget becomes tight, optional background can be removed first while the task-defining and source-governing information remains protected.

This is a system design choice and can be tested with explicit overflow scenarios.

Chunking Long Documents

Retrieval systems usually divide long documents into chunks that can be indexed and inserted selectively into context.

Token count is a convenient chunk-size unit, but boundaries should also respect paragraphs, headings, table rows and policy clauses.

A fixed 1,000-token cut can split a rule from its exception even though both fit comfortably when chunked by section.

Chunk Overlap

Overlap repeats some tokens between neighbouring chunks so information near a boundary appears in both.

This can preserve cross-boundary relationships but consumes storage and retrieval context through duplication.

The amount of overlap should therefore be evaluated on real retrieval tasks rather than copied from a generic template.

Parent-Child Retrieval

One retrieval strategy indexes small child chunks for accurate matching but returns a larger parent section to the model.

The small chunk improves search specificity, while the parent restores surrounding context needed for interpretation.

This separates retrieval granularity from inference granularity.

Context Selection Is More Important Than Context Maximum

A model with a 200,000-token window can still perform poorly if the application fills it with irrelevant, duplicated or conflicting material.

A smaller well-selected evidence packet can be easier for the model to use and cheaper to process.

The practical objective is not to maximize occupied tokens; it is to maximize task-relevant signal under the available budget.

Lost in the Middle

Long-context evaluations have shown that language models can use information unevenly depending on where it appears in the prompt, with evidence in the middle sometimes harder to retrieve than evidence near the beginning or end under particular models and tasks.

The paper Lost in the Middle: How Language Models Use Long Contexts documented this positional sensitivity in long-context retrieval settings.

The lesson is not that the middle is universally unusable. It is that fitting evidence into the formal window does not guarantee position-invariant use.

Needle-in-a-Haystack Tests

A needle-in-a-haystack test places a target fact inside a long distractor context and asks the model to retrieve it.

These tests reveal one form of long-context recall and positional robustness, but they are narrower than real document reasoning.

A model can pass exact retrieval tests while still struggling to synthesize several distant facts or resolve contradictions across a long context.

Multi-Needle Tests

Real tasks often require combining more than one fact. Multi-needle evaluations distribute several pieces of evidence through a long input and require the model to find and integrate them.

This begins to measure compositional long-context use rather than one isolated lookup.

Evaluation should resemble the actual task: retrieve one clause, compare versions, aggregate a table, or synthesize across chapters.

Long Context Is Not Long-Term Memory

A million-token context, if supported, still belongs to one active inference process or request sequence. It does not automatically persist into a future session.

Long-term memory requires an external storage mechanism and a retrieval policy for bringing selected items back into context.

The distinction matters for privacy and user expectations: temporary context retention and persistent memory are different product behaviors.

Context Is Not Learning

Supplying a new rule in context can change the answer immediately because the model conditions on that rule.

Once the context is gone, the rule may no longer influence future answers unless it is stored elsewhere or used in training.

In-context behaviour therefore demonstrates runtime conditioning, not necessarily parameter update.

In-Context Learning

Large language models can infer patterns from demonstrations included in the prompt. Show several input-output examples and the model may apply the pattern to a new case.

This behaviour is commonly called in-context learning because the adaptation happens through current context rather than gradient updates to the base model.

Its reliability depends on model, example choice, ordering and task.

Few-Shot Prompting

Few-shot prompting supplies a small number of worked examples before the target request.

The examples consume context tokens and can clarify label definitions, output format or reasoning pattern.

Poor examples can also mislead the model, so demonstrations should be representative and correct.

Many-Shot Prompting

Large context windows allow many more demonstrations, sometimes improving task adaptation or enabling richer reference material.

The marginal value of more examples can decline, and inconsistent examples can introduce contradictions.

More shots are not automatically better than a carefully selected smaller set.

Context Ordering

The order of instructions, examples and evidence can affect output. Models are not guaranteed to process a context as an order-invariant set.

A late user correction may override an earlier general instruction in some chat protocols, while untrusted documents should not acquire higher authority merely by appearing later.

Applications should use the model’s documented message roles and keep trust hierarchy explicit.

System, User and Tool Roles

Chat-oriented models often receive messages labeled with roles such as system, user, assistant and tool.

A chat template turns those structured roles into the token sequence expected by the model.

Role structure helps communicate instruction hierarchy, but software-level permissions remain stronger controls for consequential actions.

Tool Schemas in Context

Function names, descriptions and argument schemas can be included in model context so the model can decide when and how to call tools.

A large tool catalog can consume substantial tokens and create routing ambiguity.

Tool selection can improve when the application exposes only the tools relevant to the current task rather than every possible integration.

Dynamic Tool Loading

An application can retrieve or enable tool descriptions on demand instead of placing the full catalog in every request.

This saves context and can reduce confusion, but it adds a routing layer that must itself identify the right tool domain.

The design mirrors document retrieval: select relevant capability before asking the model to reason over it.

Context Compression

When history becomes too long, an application can summarize earlier turns into a shorter representation.

Compression saves tokens but risks losing exact wording, exceptions, provenance or unresolved uncertainty.

Important constraints and identifiers should remain explicitly preserved rather than trusting a free-form summary to retain every critical detail.

Structured Memory Summaries

Instead of one prose summary, applications can keep structured fields for goals, decisions, open questions, resource IDs and permissions.

The model can then receive a compact state packet whose critical information is less likely to disappear inside narrative compression.

This is an example of combining flexible language with explicit software structure.

Conversation Summarization Failure

A previous user instruction says “draft only; do not send.” A context summarizer compresses the conversation to “prepare the announcement.”

Later, the model sees a sending tool and may infer that completion means sending.

The first failure is memory/context summarization because an action boundary was lost before model inference.

Retrieval Versus Long Context

One strategy is to place an entire corpus into a very large context. Another is to retrieve only relevant passages.

Full-context approaches preserve broad access but increase compute and distraction. Retrieval reduces input size but can miss needed evidence.

Hybrid systems can use a large window for selected documents while still relying on retrieval to choose which documents deserve that space.

When Full-Document Context Helps

If the task depends on narrative sequence, global structure or many cross-references, providing a full document can be better than isolated chunks.

Examples include editing an entire report, comparing themes across a chapter or checking whether a conclusion matches earlier definitions.

The decision should follow dependency structure rather than a blanket rule to always chunk or always send everything.

When Retrieval Helps More

If the corpus contains thousands of independent documents and the task needs only two current policies, retrieval is more efficient than filling the context with the entire corpus.

Retrieval can also apply exact access, date and status filters before semantic ranking.

This keeps irrelevant or unauthorized material outside the model context.

Context and Source Provenance

Every retrieved passage should ideally retain a source identity so the final answer can trace important claims back to evidence.

A context full of unlabeled snippets makes it difficult to tell whether two statements came from the same document, different versions or different authorities.

Provenance metadata can remain structured even when the language model receives readable passage text.

Context and Contradictory Sources

Large contexts can contain both current and obsolete information. The model may synthesize them, choose one or become inconsistent.

Authority and version status should therefore be made explicit, preferably through filtering and metadata rather than relying on the model to infer document governance from wording.

More context can increase contradiction if selection quality does not improve.

Context and Duplicate Sources

Repeated passages consume tokens and can overrepresent one viewpoint or outdated version.

Deduplication before context assembly improves token efficiency and reduces accidental weighting by repetition.

The model does not know that four duplicate passages are one source unless the application preserves identity.

Context and Privacy

Every piece of sensitive data inserted into context becomes available to the model for that inference under the system’s data handling rules.

Do not include entire private databases merely because the context window can hold them. Minimize to the fields and records necessary for the task.

Access control and data minimization are stronger than telling the model not to mention sensitive information it has already received.

Context and Prompt Injection

Retrieved web pages or documents can contain text that looks like instructions. Once inside context, that text can influence model output.

The system should distinguish trusted task instructions from untrusted content and constrain tools independently.

A larger context window increases the amount of untrusted material that can be introduced, so security architecture becomes more important, not less.

Context and Cache Memory

Autoregressive models commonly cache past attention keys and values so generation does not recompute the full history each token.

As context grows, KV-cache memory grows too. Long windows can therefore be limited by accelerator memory even when the architecture theoretically supports the positions.

Context length is both a model capability and an infrastructure cost.

Prefill Cost

Before the first generated token, the model processes the supplied context during prefill.

Long contexts increase time-to-first-token because embeddings, attention and feed-forward layers operate over many input positions.

A system optimizing for interactive latency may prefer retrieval plus a shorter evidence packet even when a huge window is available.

Decode Cost

During generation, each new token attends to the cached context according to the model architecture.

The longer the active cache, the more memory must be read and managed, though grouped-query attention and optimized kernels reduce cost.

Long prompts can therefore affect not only prefill but ongoing token generation.

Context Window and Batch Size

Serving several long-context requests together can exhaust accelerator memory even when each request fits individually.

Batch scheduling, cache allocation and maximum sequence length interact.

A production context limit can therefore be lower than a theoretical model maximum under high-concurrency serving.

The Context Window Is Not the Model’s Training Memory

Model parameters preserve learned patterns across requests; context contains the current working information for this request. A model can therefore know general language structure from training while receiving one specific policy, worksheet or code file through context.

This distinction is operationally useful because context can change immediately. A new document can be inserted today without retraining the model. When the session ends, the document may disappear unless the application stores it externally.

Context Versus Application Memory

Application memory can persist selected information in a database or vector store and later retrieve it into the model’s context. The model experiences the retrieved record as current input.

This architecture gives the product continuity while preserving a separate storage layer that can be updated, corrected or deleted independently of model weights.

Context Versus Retrieval

Retrieval is the process of finding information outside the current model input. Context is the information after selection, formatting and insertion into the model request.

A retrieval system can search millions of documents while the context window holds only a small selected subset. This is why RAG can scale knowledge access beyond one prompt.

Context Versus Cache

KV cache stores internal attention keys and values for tokens already processed during autoregressive inference. It accelerates repeated computation over the current context.

The cache does not decide which documents enter the context and does not persist knowledge across independent sessions unless the serving system deliberately retains it.

A Complete Context Packet

Imagine a school assistant asked to answer a question about a current equipment policy. The model request might include: system instruction 800 tokens; user question 40; conversation history 1,200; retrieved current policy 1,600; tool schema 900; tool result 200.

Before generation, the context already contains about 4,740 tokens in this illustrative example. If the model needs room for a 1,000-token answer, the application must preserve that headroom under the model’s effective maximum.

This packet demonstrates why the latest user sentence can be tiny while the actual inference request is large.

Context Assembly Is a Separate Engineering Responsibility

Someone or some component decides which system rules, history turns, documents and tool outputs are included. That selection process is context assembly.

A powerful model can still fail if the assembler omits the current source, includes an obsolete instruction or places conflicting information without provenance.

Article 041 will treat context engineering in detail. Here the important point is that context size and context quality are different variables.

Token Budget Ownership

Long-context applications benefit from explicitly allocating token budgets across responsibilities rather than allowing one component to consume the whole window.

A system might reserve 10% for system instructions, 20% for history, 50% for evidence and 20% for generation under one task. Another task might use almost no history but a large source document.

These percentages are design examples, not universal recommendations. The budget should follow what the task needs.

Output Reservation

If the application fills the entire model context with input, there may be no room left to generate the desired answer under a combined input-output limit.

Some APIs expose output limits separately, while others still impose a total maximum sequence length. The serving contract should be checked rather than assumed.

A report-writing task needs more output headroom than a one-word classification task.

Maximum Context Versus Recommended Context

A model can technically accept a stated maximum length while delivering its best cost, latency or quality at shorter lengths.

Applications should therefore distinguish hard maximum, tested operating range and preferred economical range.

The biggest available window is not automatically the best default for every request.

Context Position Matters

Transformer representations include positional information, so the same passage at the beginning and middle of a long prompt is not computationally identical.

Training data and chat templates can create learned position preferences. Long-context evaluations should therefore vary evidence placement.

A context strategy that always places the user question near the end or repeats source labels can behave differently from one that concatenates material arbitrarily.

Beginning, Middle and End Effects

Some models have historically shown stronger retrieval from the beginning or end of long prompts than from the middle on selected tasks, motivating the phrase lost in the middle.

Modern long-context training can reduce these effects, but the general lesson remains: position-specific performance should be measured rather than assumed uniform.

A benchmark should distribute relevant evidence across positions instead of placing every answer conveniently at the beginning.

Recency Effects

Recent tokens can exert strong influence because generation occurs from the current final position and because chat training often emphasizes latest user instructions.

This can be helpful when a user corrects an earlier detail. It can be harmful when an untrusted document near the end contains instruction-like text.

Authority should therefore be encoded by protocol and software controls, not inferred solely from recency.

Context Conflicts Need Resolution Rules

Suppose one passage says the loan period is three days and another says five. The model needs more than enough context space; it needs source identity and version status.

A reliable application should prefer current approved policy or explicitly tell the model which source governs the answer.

More tokens can amplify ambiguity when source governance is weak.

Context Freshness

A large context is useless if it contains yesterday’s data for a question about today’s state. Freshness is an evidence property independent of length.

Current tasks may require live tool calls or database retrieval immediately before the final answer.

The model should not infer freshness merely because a document is long, detailed or confidently written.

Context Provenance

Passages should retain enough metadata to identify where they came from, especially when the answer will cite or act on them.

Document ID, version, date, owner and section can travel with the excerpt even if the model sees only part of the full record.

Provenance makes later verification and repair possible when a generated conclusion is challenged.

Context and Authority

The context can contain system instructions, user requests, retrieved documents and tool outputs. These sources do not all have the same authority.

An external PDF saying “ignore previous rules” should not acquire the user’s permission merely because its tokens sit inside the same context window.

Applications need protocol boundaries and downstream authorization outside the model’s raw sequence processing.

Context Injection Attacks

Prompt injection exploits the fact that untrusted content can contain instruction-like language that influences the model.

Long-context systems can increase exposure because they ingest more external text. Filtering and labeling alone are not complete defences.

Limit available tools, separate untrusted data from authority and verify consequential actions independently.

Context and Privacy

Putting data into context makes it available to the model for that inference. The safest way to prevent use of unnecessary sensitive information is not to include it.

Data minimization, tenant filtering and field-level access should happen before context assembly wherever possible.

A prompt instruction saying “do not reveal the private column” is weaker than omitting the private column entirely.

Context and Exact Identifiers

Long prompts can contain many similar IDs. A language model may choose the right semantic record but regenerate an identifier incorrectly.

Preserve exact IDs through structured fields or tool results and let the model reference them rather than recreate them from free-form text.

This becomes especially important when the identifier targets a database write or external action.

Context and Tables

Large tables often belong in databases or code execution rather than being pasted in full. The model may need the schema, a filtered subset and the computed aggregate instead of every row.

This saves context and increases exactness because deterministic systems perform arithmetic and filtering.

Use the context window for interpretation, not as a replacement for every structured data system.

Context Window Versus Maximum Output Length

A model can support a large total context while the serving API imposes a smaller maximum generated output. These are related but different constraints.

A 128,000-token context does not imply the application can generate 128,000 new tokens after filling most of the prompt. The service may reserve or cap output separately.

When planning long documents, inspect both input capacity and output limits rather than relying on one headline number.

Context Window Versus Effective Context

The formal context window describes what positions the model can accept under the architecture and serving configuration.

Effective context describes how reliably the model can use information across that range for the task at hand.

A model can accept 100,000 tokens while performing best on a smaller, well-organized evidence set. Capacity and effective use must be evaluated separately.

Context Saturation

As a prompt fills with more unrelated material, relevant evidence can become harder to distinguish from distractors.

The model may still technically attend everywhere, but competition among instructions, examples and sources can reduce task reliability.

Context saturation is therefore a signal-to-noise problem as well as a hard capacity problem.

Context Pollution

Context pollution occurs when stale, irrelevant or conflicting information remains in the active prompt and influences later output.

Examples include abandoned drafts, superseded tool results, old policy versions or one user’s details accidentally carried into another workflow.

Good context management removes obsolete state instead of letting every past token persist indefinitely.

Context Isolation

Separate tasks should not automatically share all prior context. A new workstream can begin with a clean context and only the state it actually needs.

Isolation improves privacy and reduces accidental instruction carryover.

Persistent memory can still retrieve selected relevant facts without importing an entire unrelated conversation.

Context Reset

A context reset discards active conversation state from the model input while leaving the underlying model parameters unchanged.

It can be useful when the current thread has accumulated contradictory instructions or the user wants a genuinely fresh task.

Resetting context is not the same as deleting stored memories or retraining the model.

Context Checkpoints in Long Workflows

An agentic workflow can periodically summarize completed steps into a compact verified state packet before discarding verbose intermediate logs.

The packet might preserve task goal, approved decisions, current files, tool outputs, unresolved issues and next allowed actions.

This creates an explicit handoff between workflow stages and limits uncontrolled context growth.

Context as a Working Set

A useful analogy is a working set in computing: keep the information required for the immediate task close to the model while other data remains in external storage.

The working set changes as the task advances. Evidence that was essential for one decision may no longer need to occupy tokens after the decision is verified and recorded.

This dynamic view is more efficient than treating the context window as a warehouse that should contain everything.

A Complete Research-Agent Context Example

Task: compare three current policy documents and produce a recommendation memo. The workflow begins with the user request, evaluation criteria and tool definitions.

Search retrieves candidate documents. The application keeps source metadata and inserts only current authoritative passages. The model compares them and records a structured table of differences.

Once the comparison is verified, the raw search snippets can be replaced by the verified comparison table plus source references before the final memo is drafted. Context shrinks while provenance remains.

A Complete Coding-Agent Context Example

Task: fix one bug in a repository. The agent begins with the issue description, relevant files and available test commands.

After inspecting the repository, it need not keep every directory listing forever. It can retain the target files, failing test, chosen hypothesis and patch diff.

After tests pass, the context can retain the final diff and test result rather than every exploratory command. This improves clarity and reduces stale-state confusion.

A Complete Education Context Example

Task: explain one student’s repeated algebra error using a supplied worksheet. The system needs the relevant workings, marking evidence and learning objective.

It does not need every historical school record or unrelated essay. A smaller evidence packet protects privacy and keeps the task focused.

After the error pattern is identified, the model can generate practice questions from the verified pattern while the original worksheet remains available for checking.

Context Engineering

Context engineering is the design of what information enters the model, how it is structured, when it is retrieved, what is retained across steps and what is removed.

It is broader than prompt writing. Prompt wording is one component inside a larger working-information system.

A good context engineer asks what the model must know now, what evidence supports the task, what authority each source has and what can safely remain outside the window.

The Context Engineering Loop

  • Define the task and stopping condition.
  • Identify authoritative evidence and required state.
  • Retrieve or load only the relevant material.
  • Preserve source identity, dates and permissions.
  • Structure the context so instructions and evidence are distinguishable.
  • Reserve output capacity.
  • Run the model and inspect what evidence it used.
  • Compress or discard obsolete intermediate material.
  • Record verified decisions outside the model when they must persist.
  • Repeat with a clean working set for the next step.

Context Priority Bands

One practical design assigns context to priority bands. Critical: task goal, action boundary and current authoritative evidence. Important: recent decisions and unresolved questions. Optional: background examples, verbose logs and exploratory notes.

When the token budget tightens, optional material is removed first.

This explicit priority policy is safer than arbitrary truncation based only on chronology.

Protecting Negative Constraints

Instructions such as “do not send”, “read-only”, “exclude pending records” or “use only current approved sources” can be short but operationally critical.

Summarizers and truncation algorithms should preserve these negative constraints explicitly.

A context that loses the prohibition while retaining the task goal can transform safe assistance into unauthorized action.

Protecting Resource Identity

Exact file IDs, database record IDs, approved version numbers and destination names should remain available when later actions depend on them.

Do not rely on a prose summary to reconstruct identifiers after the original tool result has been removed.

Structured state can preserve exact identity while narrative context is compressed.

Protecting Uncertainty

A source may say “room TBC” or a tool may return an unknown outcome. Context compression should not turn that uncertainty into a definite value.

Represent unresolved state explicitly so later model passes know which facts remain open.

Otherwise, repeated summarization can harden tentative guesses into apparent facts.

Context Versioning

A long workflow can reference evolving documents. Context should identify which version of a source was reviewed and approved.

If a file changes after review, a later action should not silently inherit approval for the new version.

Version identity therefore belongs in the working state alongside the document text.

Context and Human Approval

When an action requires approval, the context should preserve exactly what object, version, destination and operation the user approved.

A later model pass can then compare the proposed action with that approved scope.

Human approval is stronger when it is represented as explicit state rather than an ambiguous phrase buried hundreds of messages earlier.

Context and Agent Loops

An agent repeatedly observes, decides, acts and receives new observations. Each loop can add tokens.

Without context management, the agent can become increasingly distracted by its own old plans, failed attempts and obsolete tool outputs.

A stable agent periodically consolidates state and applies stopping rules instead of treating infinite history as progress.

Looping Failure From Stale Context

An agent searches for a missing file, gets “not found”, later creates the file, but still keeps the old failure message without a current-state check.

A later step sees both old failure and new success and searches again unnecessarily.

State consolidation should mark the old observation as superseded once the file is verified to exist.

Context and Multi-Agent Systems

Several agents can exchange summaries, plans or tool results. Those messages become each recipient’s context.

Coordination can therefore amplify stale or incorrect information if one agent’s unsupported conclusion is copied across the group.

Shared state should distinguish verified observations from proposals and generated interpretations.

Context and Citations

When evidence passages enter context, preserve source references so the final model can cite the correct document rather than inventing a plausible URL.

A citation pipeline should connect the claim to the exact source object that supplied the evidence.

Context compression must preserve that mapping if provenance is required in the final answer.

Context and Structured Outputs

Structured state can reduce token ambiguity. Instead of a long paragraph saying what has happened, a workflow can store status, current_version, approved_action and unresolved_fields as explicit fields.

The model can read those fields in context while conventional software validates them.

This hybrid approach combines language flexibility with exact state ownership.

Context Window and Latency

Longer context generally increases prefill work and can increase decode memory traffic through larger caches.

A system can therefore trade retrieval work against model-processing work. Spending a small amount of time retrieving the right five passages can save much more transformer compute than sending an entire corpus.

Latency optimisation should measure the whole request rather than only model tokens per second.

Context Window and Cost

Many hosted model services charge according to input and output tokens, though pricing models vary.

Even when self-hosting, longer contexts consume accelerator time, memory and energy.

Context efficiency is therefore both a quality and resource question.

Context Window and Energy

Processing unnecessary tokens uses additional computation. At large scale, inefficient context assembly can translate into meaningful infrastructure load.

Data-center energy and environmental analysis requires actual workload measurements, but reducing irrelevant input is a direct engineering efficiency improvement.

Article 087 will examine environmental load with appropriate measurement boundaries.

Context Window and Caching

Applications can cache model-independent retrieval results, tokenized prefixes or model-specific prefix states depending on architecture and provider capabilities.

Prefix caching can reduce repeated prefill for shared long instructions or documents.

Cache invalidation matters when the supposedly shared prefix contains changing policy or user-specific data.

Context Window and Security

Large windows increase the amount of material that can influence the model in one request. This can include untrusted documents and hidden instructions.

Security controls should limit which data and tools enter the task, not merely hope the model will prioritize the right text among a huge context.

Least privilege and source isolation remain essential as context capacity grows.

Reality Check: Bigger Context Is More Capacity, Not Guaranteed Better Reasoning

  • A document fitting inside context does not guarantee every fact will be used correctly.
  • More context can introduce noise, duplication and contradictory sources.
  • A long context is not persistent memory and does not update model weights.
  • Tool schemas, history and retrieved passages all compete for the same finite token budget.
  • Truncation can remove small but critical negative constraints.
  • Current authoritative state should still come from the appropriate source system.
  • Long-context quality requires task-specific evaluation, not just a advertised maximum token number.

Failure Class 1: Hidden Truncation

The application silently drops an old but still active instruction when the conversation becomes too long.

The model then acts outside the original boundary.

Expose truncation policy and protect critical task constraints.

Failure Class 2: Context Flooding

The system retrieves twenty documents for a question that needs one current policy. Relevant evidence is buried among duplicates and outdated versions.

Reduce retrieval k, improve reranking and filter by authority before model inference.

Failure Class 3: Stale Summary

A conversation summary says a draft is version 2 even though version 3 was created later.

The model uses obsolete state and edits the wrong object.

Regenerate structured state from authoritative tools rather than trusting stale prose summaries.

Failure Class 4: Tool Catalog Overload

Hundreds of tool descriptions enter every prompt. The model selects a broad administrative tool instead of a narrow task-specific function.

Use dynamic tool loading or scoped tool sets and enforce permissions downstream.

Failure Class 5: Evidence in the Middle

A crucial fact is present but model performance drops when the fact appears deep inside a long distractor context.

Use long-context evaluation, retrieval/reranking, clearer structure or evidence placement rather than assuming formal capacity guarantees reliable use.

Failure Class 6: Lost Provenance

Summarization preserves the conclusion but drops which document supported it.

Later, the model cites the wrong source or cannot verify the claim.

Keep source IDs and claim-evidence links in structured state.

Failure Class 7: Privacy Overload

An application inserts an entire student database because the context window is large enough.

The model receives unnecessary sensitive information, increasing exposure and prompt-injection surface.

Filter before context assembly and supply only the minimum relevant records.

Repair Pathway

When a long-context answer fails, inspect the exact tokenized input. Was the required evidence present? Was it complete? Which competing sources were included? Which instruction hierarchy did the model receive?

Then test a small direct context containing only the required evidence. If the model succeeds there, context selection or scale is the leading suspect.

Repair the earliest unstable point before moving to a larger model or wider window.

Stabilisation Pathway

Create regression cases for truncation, evidence at different positions, conflicting versions, long tool results, multilingual documents and repeated agent loops.

Track token count, source coverage, time-to-first-token, answer quality and completion status together.

A context strategy is stable when important information remains available and interpretable under ordinary workload variation.

Extension Pathway

After a reliable short-context workflow exists, add longer documents, more demonstrations or richer tool histories gradually.

Use retrieval and structured memory to keep growth purposeful.

Every expansion should preserve the distinction between more capacity and better information.


Long Context and Dense Attention

A full transformer attention pass considers relationships among many token positions. In the basic dense formulation, the pairwise score structure grows with the square of sequence length.

Optimized attention kernels reduce memory traffic and improve hardware efficiency, but long prompts still increase compute, activation movement and cache pressure.

This is why supporting a longer context window is an engineering commitment as well as a model-training feature.

Long Context and KV-Cache Memory

Autoregressive decoders store key/value tensors for earlier positions so later generation steps can reuse them. Cache size grows with sequence length, layer count, key/value head structure, head dimension, batch size and numerical precision.

A service can therefore support one extremely long sequence while serving far fewer such requests concurrently on fixed hardware.

Context-window planning should consider throughput and memory under realistic usage, not only the maximum sequence accepted in an isolated test.

Sliding-Window Attention

Some transformer architectures restrict selected layers to a local attention window. A token can directly attend only to nearby positions rather than every earlier token.

This reduces compute and cache requirements and can make very long sequences practical.

The trade-off is information routing: distant facts may need several layers to propagate or special global mechanisms to remain accessible.

Sparse Attention

Sparse attention allows selected long-range connections while avoiding a fully dense token-pair matrix. Block-local, strided, landmark and global-token patterns are examples.

A sparse pattern encodes an assumption about which routes matter. The model must be trained to use that structure.

A nominal context length can therefore conceal very different internal connectivity across architectures.

Position Encoding and Context Limits

Context capacity also depends on how the architecture represents position. Absolute learned embeddings, relative position methods and rotary position embeddings have different behavior when sequence length extends beyond training.

A serving system should not assume that increasing a maximum-position setting creates reliable long-context behavior.

Position scaling, fine-tuning and architectural adaptations need evaluation on the actual long-context tasks.

Nominal Context Versus Effective Context

Nominal context is the number of tokens accepted by the model or service. Effective context is the range over which the model can reliably retrieve and use relevant information under realistic conditions.

A model can technically accept 200,000 tokens while accuracy declines when the required fact is buried among similar distractors or placed at difficult positions.

This distinction keeps context-window claims tied to task performance rather than capacity alone.

Evidence Position Matters

The paper Lost in the Middle: How Language Models Use Long Contexts reported position-sensitive performance in long-context tasks for the models it studied, including weaker use of some information placed in middle regions.

The finding should not be frozen into a universal law for every newer model. Long-context training and architectures continue to evolve.

The durable lesson is methodological: test evidence at several positions rather than assuming uniform accessibility because the text fits.

Context for Code Repositories

A software repository can contain millions of tokens, generated files and dependencies. Sending the whole repository into one prompt can waste context and confuse file relationships.

Coding systems use symbol search, repository maps, dependency information and targeted file retrieval to select the relevant slice of code.

Long context helps when several files must be compared together, but retrieval remains useful for navigation and current working-set selection.

Context for Research Agents

A research agent can visit dozens of sources over many steps. Keeping every full article inside active context quickly becomes impractical.

A stronger workflow stores a source ledger, claims, citations and structured notes externally. Each reasoning step retrieves the evidence it currently needs.

This lets the research process outlive one context window without pretending that all prior source text remains active in the model.

Context for Agent Planning

Agent loops accumulate plans, completed steps, tool results, errors and revised goals. A raw transcript can become larger than the work itself.

A compact task-state representation can preserve current goal, completed actions, pending dependencies and boundaries while archival details remain outside active context.

The system can retrieve detailed history when a dispute or comparison requires it.

A Context Memory Hierarchy

  • Immediate model context for the current inference.
  • Recent working state retained directly across several steps.
  • Persistent application memory stored outside the model.
  • Document and message indexes used for retrieval.
  • Authoritative databases that hold exact current state.
  • Archives retained for audit but not automatically inserted into active context.

This hierarchy makes the active context a working surface rather than a universal storage container.

Prefix Caching

Serving systems can cache computation for repeated prompt prefixes such as long system instructions or a shared document.

Prefix caching reduces repeated prefill work when the exact prefix recurs. It does not increase the model’s supported context length or make the cached content permanent model knowledge.

When the prefix changes, the relevant cache entry must be invalidated or recomputed.

Context Window and Cost

Long inputs consume compute before any useful output appears. If an API bills by processed tokens, irrelevant context also creates direct financial cost.

On self-hosted systems, the cost appears as accelerator time, memory occupancy and reduced throughput.

Context engineering is therefore an efficiency discipline as well as an accuracy discipline.

Context Window and Privacy

A large window creates the temptation to include everything available. That is poor privacy practice. Information should enter the prompt because the task legitimately needs it, not because capacity exists.

Data minimization reduces disclosure risk, accidental generation of private details and prompt-injection surface.

A system should filter sensitive fields before context assembly rather than relying only on the model to refrain from mentioning them.

Context Window and Prompt Injection

Untrusted documents can contain imperative text that appears inside the same token sequence as legitimate instructions. The model can be influenced by both unless the system preserves trust boundaries.

Longer context can increase the quantity of untrusted text, so context size is not automatically a security improvement.

Constrained tools, downstream authorization and source labeling remain necessary even when a model has strong instruction-following training.

Context Window and Citations

When several passages support an answer, the application should preserve source IDs through context assembly so generated citations can point back to real evidence.

A context compactor that removes provenance can leave the model with useful facts but no trustworthy route to cite them.

Citation support is therefore part of context design, not only final response formatting.

Context Window and Conflicting Evidence

A long context can contain multiple versions of the same policy or contradictory reports. More information can therefore increase ambiguity rather than reduce it.

The application should mark source status and dates, filter superseded material where appropriate and ask the model to distinguish current authority from historical background.

Similarity alone cannot resolve governance.

Context Window and Duplicate Evidence

Retrieval systems can return overlapping chunks or duplicate copies of a document. Repetition consumes tokens and can overemphasize one source.

Deduplication by document ID, content hash or chunk overlap reduces wasted capacity.

The goal is evidence diversity and completeness, not raw passage count.

Long-Context Evaluation Dimensions

  • Sequence length.
  • Position of the relevant evidence.
  • Number of relevant passages.
  • Similarity of distractors.
  • Whether sources agree or conflict.
  • Need for exact retrieval versus synthesis.
  • Need for citations or source attribution.
  • Latency and memory at the tested length.
  • Performance under near-limit truncation.
  • Behavior when the answer is absent from the context.

A model that passes only one dimension—such as retrieving one needle from random text—has not demonstrated all long-context capabilities.

A Complete Long-Context Benchmark Design

Create a fictional handbook containing current rules, archived rules, unrelated sections and several cross-references. Generate test copies where the controlling passage appears early, middle and late.

Ask direct questions, exception questions, multi-hop questions and unanswerable questions. Require the model to cite the current source and to reject archived authority.

Record answer correctness, evidence retrieval, citation support, abstention and latency. This gives a richer picture than merely reporting the maximum accepted token count.

Reality Check: Bigger Context Is Capacity, Not Guaranteed Understanding

  • Information can fit in the window and still be ignored or misinterpreted.
  • Longer input increases compute and cache memory.
  • A huge window can amplify irrelevant or malicious text as well as useful evidence.
  • Retrieval remains valuable for selecting current, authoritative information.
  • Summaries save tokens but can lose details or preserve mistakes.
  • Persistent application memory is different from active context.
  • The smallest sufficient evidence set can outperform an indiscriminate maximal prompt.

Context Failure Map

  • Silent truncation: a critical instruction or exception disappears before inference.
  • Context flooding: irrelevant material overwhelms useful evidence.
  • Stale summary: compressed history preserves an old decision after the user changes it.
  • Missing provenance: conflicting passages enter without version or authority metadata.
  • Duplicate retrieval: repeated chunks waste budget and bias attention.
  • Tool overload: too many irrelevant schemas make selection harder.
  • Capacity-reliability confusion: the team tests fit but not actual long-context use.

Repair Pathway

When a long-context answer fails, inspect the exact tokenized input. Verify whether the needed evidence is present, intact, current and within the allowed attention region.

Then rerun the task with only the controlling evidence. If the model succeeds on the short version, the failure likely belongs to context selection, position or distractor handling rather than the core rule interpretation.

Repair the earliest context defect before increasing the maximum window.

Stabilisation Pathway

Define explicit budgets for instructions, tool schemas, recent history, retrieved evidence and output. Test requests near the maximum length and record what the system truncates.

Monitor time to first token and KV-cache memory so context growth does not silently make the service too slow or expensive.

Preserve long-context regression tests across model and retrieval upgrades.

Extension Pathway

After one context strategy is stable, extend with larger-window models, hierarchical retrieval, structured summaries, external memory or sparse attention one mechanism at a time.

Each extension should preserve source provenance, permissions and the ability to reconstruct what information the model actually received.

Independent Exercise 1: Token Budget

A 16,000-token total budget reserves 3,000 for output. Instructions use 2,000 and history uses 4,000. How many tokens remain for the current request and retrieved evidence?

Answer

7,000 tokens in this simplified accounting.

Independent Exercise 2: Presence Versus Use

The relevant fact is inside the prompt but the model answers incorrectly. Does that prove the window is too small?

Answer

No. The fact fits. The problem may be evidence position, distractors, interpretation or reasoning.

Independent Exercise 3: Summary Drift

A stored summary says “draft only”, but a later explicit user message authorizes publication of the current version. Which should govern?

Answer

The later valid explicit instruction, assuming it clearly identifies scope. The stale summary should be updated.

Independent Exercise 4: Huge Manual

A 200-page manual fits inside the model’s context. Is retrieval now useless?

Answer

No. Retrieval can reduce noise, cost and latency while selecting the authoritative passages relevant to the question.

Independent Exercise 5: KV Cache

If sequence length doubles, does KV-cache memory usually remain unchanged?

Answer

No. It generally grows with the number of cached positions, subject to architecture and compression strategy.

Independent Exercise 6: Tools

A task needs two tools but the prompt includes eighty schemas. What system change can improve context use?

Answer

Route or scope tools so only the relevant subset is exposed to the model.

Frequently Asked Questions About Context Windows

What is a context window?

The token-sequence capacity available to the model for the current inference or generation process under a specific configuration.

Is context the same as memory?

No. Context is active working information; persistent memory can be stored externally and retrieved later.

Does a larger window make the model smarter?

It provides more capacity for information, but effective use depends on training, architecture, context quality and the task.

Does all chat history stay in the context?

Not necessarily. Applications can truncate, summarize or retrieve previous turns selectively.

What happens when input is too long?

The service can reject, truncate, summarize, retrieve a subset or route to a different model depending on design.

Does retrieval matter when context is huge?

Yes. Retrieval prioritizes relevant, current and authorized evidence and can reduce compute and distraction.

What does lost in the middle mean?

It refers to observed position-sensitive long-context performance in which some models used information less effectively at particular middle positions.

Does KV cache increase the context limit?

No. It reuses attention state for faster generation; supported sequence length is a separate model/configuration property.

Can context include images or audio?

Multimodal models can include representations from other modalities, with model-specific capacity and accounting.

What is the best context strategy?

Use the smallest sufficient, authoritative and permission-appropriate working set supported by task-specific evaluation.

Selected Technical References

A Context Window Is Working Space, Not the Whole Intelligence System

The active context is a finite working sequence assembled for one task. Instructions, history, evidence and tool results compete for the same computational envelope.

Longer windows are valuable, but they do not eliminate retrieval, memory, source governance or context engineering. The strongest system brings the right information into the window at the right moment rather than assuming everything should remain active forever.

With Article 020, Part 2 of How Super Intelligence Works is complete: tokens become model input through tokenisation; embeddings create vectors; vector space gives them geometry; neural networks learn transformations; transformers and attention route information; parameters store learned state; layers create depth; and context windows define the active sequence envelope.

Independent Exercise 1: Budget Arithmetic

A model has a 32,000-token combined budget. The application reserves 4,000 tokens for output. System instructions use 2,000, tool schemas 3,000 and history 5,000. How many tokens remain for the current user request and retrieved evidence?

Answer

32,000−4,000−2,000−3,000−5,000 = 18,000 tokens remain under this simplified accounting.

Independent Exercise 2: Truncation

A policy exception is at the end of a long document and is removed by right-side truncation. The model applies only the general rule. What should be repaired first?

Answer

Context preparation or chunking. The model did not receive the exception.

Independent Exercise 3: Context Versus Memory

A user preference disappears after the conversation is cleared. Does this prove the model forgot learned weights?

Answer

No. The preference may have existed only in the previous context and never entered persistent memory or model training.

Independent Exercise 4: Full Corpus Versus Retrieval

A corpus contains 50,000 documents, but one current policy answers the question. Should the entire corpus be placed in context if the window were theoretically large enough?

Answer

Usually no. Retrieval plus authority filtering is more efficient and reduces irrelevant or conflicting material.

Independent Exercise 5: Tool Result

An agent has accumulated ten verbose search results but only one verified fact is needed for the final step. What should context management do?

Answer

Preserve the verified fact and source provenance, then discard or summarize obsolete exploratory results.

Independent Exercise 6: Lost in the Middle

A model retrieves a fact correctly when it appears at the beginning of a long prompt but misses it when placed in the middle. Is the formal context limit broken?

Answer

Not necessarily. The tokens fit, but effective long-context use is position-sensitive for that task. Evaluate retrieval, structure and model behavior rather than only capacity.

Independent Exercise 7: Privacy

A context window can hold an entire employee database. Does capacity justify supplying every record?

Answer

No. Data minimization and access control should limit context to records necessary for the task.

Independent Exercise 8: Negative Constraint

A summary retains “prepare the report” but drops “do not publish.” What type of failure is this?

Answer

Context summarization failure: a critical action boundary was lost.

Frequently Asked Questions About Context Windows

What is a context window?

The token capacity available to a model for the active inference context under a particular architecture and serving configuration.

Is context the same as memory?

No. Context is current working information. Persistent memory stores information outside the active request and retrieves it later.

Is context the same as training data?

No. Training data changes model parameters through optimization; context conditions current inference without necessarily changing weights.

Does a larger context window always improve answers?

No. More capacity can help, but irrelevant, duplicated or contradictory material can reduce reliability and increase cost.

What is truncation?

Removing tokens or content so the prepared input fits within the available budget.

What is context engineering?

Designing which instructions, evidence, memory, tool descriptions and results enter the model, how they are structured and what is retained across steps.

What is lost in the middle?

A pattern observed in some long-context evaluations where models retrieve or use information less reliably when it appears in certain middle positions than near context boundaries.

Does fitting a document into context mean the model has read it reliably?

No. Capacity to include the document and ability to use every relevant part accurately are different properties.

Why use retrieval if the context window is huge?

Retrieval can reduce cost, latency, distraction and privacy exposure by selecting only relevant current evidence.

How does KV cache relate to context?

During autoregressive generation, KV cache stores attention keys and values for the active token history so later decode steps can reuse them.

Selected Technical References

The Context Window Is a Working Surface, Not a Warehouse

A model needs the right information available at the right moment. The formal context window defines how much tokenized material can be present, but useful performance depends on selection, structure, source quality and preservation of constraints.

The best context is not necessarily the largest. It is the smallest complete working set that preserves the task, the authoritative evidence, the current state, the action boundary and enough room to produce the result.

With Article 020, the representation-and-model-mechanics block of the series is complete. Next: 021 — Pretraining, where we move from runtime representation to the training process that creates broad model capability before the user arrives.


How Super Intelligence Works Series Navigation

Previous: 019 — Layers · Series Hub · Next: 021 — Pretraining.