What Is a Context Window
What a Context Window Is
A context window is the maximum amount of text a language model can take into account in a single request. That includes the instructions, the conversation so far and the model’s own reply. The window is measured in tokens, the small text units models read, and a model with a larger window can consider more material at once.
The model does not remember anything outside that window. Each request is processed using only what is included in it. Anything not sent, such as a conversation from last month, is invisible to the model unless the application adds it back.
What Happens When the Window Fills Up
When the input and the expected output together would exceed the limit, the request fails or the application must shorten what it sends. Common approaches include:
- Truncation, which drops the oldest messages.
- Summarization, which replaces earlier turns with a short summary.
- Retrieval, which searches a larger store and includes only the relevant passages.
Each method has trade-offs. Truncation can lose important early details, and summaries can omit or distort facts. Retrieval depends on finding the right passages, as explained in the topic on retrieval-augmented generation.
How Context Length Affects Cost and Chatbot Behavior
Input tokens are usually billed, so a full window costs more per request. A chatbot that resends a long conversation on every turn becomes slower and more expensive as the conversation grows. The way tokens are priced determines how quickly that growth shows up on the bill.
Longer context can also change quality. Models do not always weigh every part of a long input equally, so important details placed in the middle of a large document may be used less reliably. A bigger window is therefore not a substitute for selecting the right context. When models run on your own hardware, long contexts also need more memory, as explained in the topic on VRAM.
A Support Bot That Forgets an Order Number
A store’s support chatbot answers a customer’s question about a return. Twenty messages later, it asks again for the order number. The conversation history has grown past the window limit, so the application dropped the early messages.
The fix is to keep key facts, such as the order number, in a short summary or a structured record that is always included with each request.
Frequently asked questions
Is a bigger context window always better?
Not always. A larger window allows more material, but it increases cost and can slow responses. It also does not guarantee that the model will use every part of the input well. For many tasks, sending the most relevant content is more effective than sending everything.
What is the difference between a context window and memory?
A context window is the text the model can see during one request. Memory usually means information stored outside the model, such as a saved profile or a log of past conversations, which the application retrieves and adds to the window when needed. Some products offer memory features that work this way. The model itself does not keep that information between requests by default.