A Larger Context Window Is Not an Evidence Strategy

One of the most useful advances in large language models is their ability to process increasingly long inputs.

A larger context window makes it possible to include more documents, conversation history, source code, and tool results in a single request.

But having more space doesn't mean we should fill it.

In a RAG application, for example, it's tempting to retrieve more document chunks whenever an answer is incomplete. Increasing the number of results feels like a reasonable solution because it gives the model more information to work with.

Sometimes that helps. Other times, it introduces irrelevant passages, repeated content, outdated information, and unnecessary processing costs.

The result can be a longer prompt without a more reliable answer.

The important question isn't how much context the model can accept. It's how much relevant evidence the task actually needs.

This article covers how I approach context selection, token budgeting, and evaluation when building LLM-powered applications.

1. More context doesn't always mean better reasoning

A language model's context window defines how much information it can process within its configured limits.

It doesn't guarantee that every piece of information will receive the same attention or be used correctly.

The research paper Lost in the Middle: How Language Models Use Long Contexts, first submitted in July 2023, investigated this problem through multi-document question answering and key-value retrieval tasks.

The researchers found that the position of relevant evidence could significantly affect model performance.

In many of their experiments, models performed better when important information appeared near the beginning or end of the input. Performance often dropped when that information appeared in the middle of a long context.

The study also showed that increasing the available context length didn't automatically improve how effectively the tested models used information.

These results describe the models and tasks evaluated in that research. They shouldn't be treated as a universal limitation of every modern model.

Still, the finding highlights an important engineering concern.

A document can be present in the prompt without being used correctly in the answer.

That's why I prefer testing a system's ability to use evidence rather than assuming a larger context window will solve retrieval or reasoning problems.

2. Start with the question, not the available documents

Before building a prompt, I want to understand what information the task actually requires.

Consider an internal knowledge assistant answering:

What is our current refund policy for annual subscriptions?

The knowledge base might contain hundreds of documents, including old refund policies, customer-support conversations, billing guides, and unrelated product documentation.

Sending everything to the model would create unnecessary complexity.

Instead, I would identify the relevant policy, verify its effective date, and retrieve the passages needed to answer the question.

That leads to a more focused context.

For a RAG system, I generally think about evidence selection in three stages.

Candidate retrieval: Use keyword search, semantic search, or hybrid retrieval to identify potentially relevant passages.

Ranking and filtering: Remove irrelevant or unauthorized results, handle duplicates, and prioritize the most useful evidence.

Context construction: Assemble the selected passages with source identifiers, document metadata, and enough surrounding information to preserve their meaning.

The goal isn't always to retrieve fewer passages.

Some questions genuinely require information from multiple documents.

The goal is to include enough evidence to answer the question correctly without adding material that doesn't contribute to the task.

3. Treat the context window as a budget

Every LLM request has a limited amount of space available.

That space may need to accommodate system instructions, user messages, conversation history, tool definitions, retrieved evidence, and generated output.

I prefer to make these costs visible rather than letting the prompt grow until it approaches the model's maximum capacity.

A simplified planning formula is:

Evidence budget =
    Configured context allowance
  - Instructions
  - User input and conversation state
  - Tool definitions and required results
  - Reserved generation capacity
  - Safety margin

This is a planning model, not a universal accounting formula for every API.

Actual token accounting varies by model and provider. Some systems include additional formatting overhead, multimodal inputs, or internal reasoning tokens.

For example, suppose an application is configured with a hypothetical 32,000-token allowance for one request.

It might reserve:

These numbers are illustrative, not recommended defaults.

The actual budget should come from the deployed model's limits and the application's needs.

I also avoid enforcing token limits using character counts alone.

Different languages, technical identifiers, structured data, and formatting can produce different token counts.

For production applications, I prefer model-compatible token counting or an official input-token counting API where available.

OpenAI token-counting documentation

The key is to reserve enough capacity for the model to complete its work instead of using the entire allowance for input context.

4. Remove repetition before removing useful evidence

One of the easiest ways to waste context is by including the same information multiple times.

This happens frequently in document-based RAG systems.

A document may be split into overlapping chunks. When several neighboring chunks rank highly, the retriever can return nearly identical passages.

If all of them are included, the model receives repeated information while other useful sources may be excluded.

I would address this during context construction.

A practical approach is to:

• Remove exact duplicate passages.
• Group related chunks by their original document.
• Identify significant overlap between neighboring chunks.
• Preserve source identifiers and important metadata.
• Keep enough surrounding text to avoid losing meaning.
• Prefer the most relevant evidence when the context budget is limited.

Deduplication needs to be careful.

Two passages that look similar may describe different product versions or policies with different effective dates.

Removing one without checking its metadata could discard information the model needs.

For example, an older refund policy and a current policy might differ by only one sentence.

That sentence could be the most important part of the answer.

This is why source identity, document version, and effective dates matter just as much as semantic similarity.

5. Keep evidence traceable

A reliable RAG system should be able to explain where its answers came from.

That becomes more difficult when the application combines many documents into a single block of text without preserving their identities.

I prefer keeping retrieved passages structured.

For example:

Source: refund-policy-2026
Title: Annual Subscription Refund Policy
Effective Date: 2026-01-01
Section: Refund Eligibility

Content:
[Relevant source passage]

This structure helps the application preserve the relationship between evidence and its source.

It also makes citations easier to generate and validate.

In a production system, source metadata should be maintained separately from the model's generated answer. The application should be able to verify that a cited document and passage were actually retrieved.

The distinction matters because a model can generate a convincing citation that doesn't correspond to supporting evidence.

I also treat retrieved content as untrusted input.

A document may contain instructions that are relevant as document content but should not control the application's behavior.

Keeping system instructions separate from retrieved evidence helps maintain that boundary.

The model should use retrieved documents as evidence, not as instructions that override application rules.

6. Use summaries carefully

Conversation history and tool outputs can consume a large part of the context window.

Summarization is a common way to reduce that cost.

For example, an AI assistant working through a long customer-support conversation may not need every previous message in full.

A structured summary can preserve the customer's issue, previous actions, and current status.

But summaries introduce a risk.

They can omit details that later become important.

Imagine a conversation where a customer mentions that a payment problem only occurs with one specific bank account.

If the summary reduces that to "Customer has a payment issue," the assistant may lose the information needed for the next troubleshooting step.

I prefer separating durable facts from conversational summaries.

Important identifiers, verified decisions, current states, and source references should remain available in structured storage.

A summary can help the model understand the conversation, but it shouldn't become the only record of critical information.

The same principle applies to tool results.

If an agent needs an order status, it may only require the order ID, current state, and last update time.

It probably doesn't need the entire internal API response.

Filtering unnecessary fields reduces context usage and makes the next decision easier to understand.

7. Test context size and evidence placement

Context optimization should be measured rather than based entirely on intuition.

I would start with a fixed evaluation dataset containing representative questions and known supporting evidence.

Then compare several context configurations.

For each configuration, I would measure answer correctness, evidence support, citation accuracy, latency, and token usage.

I would also include questions that require evidence from more than one source.

These cases are useful because a compact context can fail when it removes information needed to connect multiple facts.

Testing evidence position

One practical experiment is to keep the relevant evidence unchanged while moving it to different positions within a controlled context.

For example:

• Relevant passage near the beginning.
• Relevant passage near the middle.
• Relevant passage near the end.

Keep the question, model, generation settings, and total context content as consistent as possible.

If accuracy changes significantly, the application may be sensitive to evidence placement.

That result can help guide context construction, but it shouldn't lead to a universal rule that important evidence must always appear first.

The right ordering can depend on the model, document structure, and task.

I would test the actual production configuration before changing it.

8. Balance quality, latency, and cost

Longer contexts can increase processing time and cost, depending on the model and provider.

But the cheapest prompt isn't necessarily the best prompt either.

A context that's too small may omit important evidence, produce incomplete answers, or require additional retrieval attempts.

I prefer evaluating three dimensions together:

Quality: Does the model answer correctly and support its claims with evidence?

Performance: How long does the complete request take, including retrieval and generation?

Cost: How many input and output tokens are consumed, and what is the resulting operating expense?

Average latency is useful, but the 95th percentile can reveal slow requests that affect the user experience.

I also monitor how often the model gives unsupported answers or fails to use evidence that was correctly supplied.

If a larger context improves difficult questions without creating unacceptable latency or cost, it may be the right choice.

If the additional content only adds repetition, a smaller and better-ranked evidence set is usually easier to maintain.

The objective is not to minimize tokens at any cost.

It's to use the available context effectively.

Final thoughts

A larger context window is a useful capability, but it isn't a replacement for good evidence selection.

In production LLM applications, reliability depends on retrieving the right information, preserving its meaning, controlling how it enters the prompt, and checking whether the model actually uses it.

My approach is to start with the task, select relevant evidence, remove unnecessary repetition, preserve source references, and test different context configurations against real questions.

Sometimes a larger context is necessary.

Other times, better retrieval and a smaller, carefully constructed prompt produce more consistent results.

The decision should come from evaluation rather than the assumption that more information is always better.

A reliable AI system doesn't need to show the model everything. It needs to provide the right evidence, at the right time, in a form the model can use effectively.