Скачать или смотреть Does the size of your context window matter?

If you've been using ChatGPT, you may have noticed that during long conversations, your AI assistant may forget the earlier part of your conversations.

This is because LLM's have a context window that is essentially the length of text that LLM can digest at any given time. GPT-4 currently has a context window of 32k tokens (only available through their API) which means you can have about 20k words being sent to and received from a LLM.

Anthropic recently announced a 100k (75k words) context window for their AI assistant Claude. Imagine being able to feed a LLM a short book and giving you a summary in seconds.

A large context window also means the AI assistant can remember more information from your conversation.

But keep in mind that a larger context window isnt always the best solution. For one, there is an increased cost with using large prompts (like prompts that include a short book) and the accuracy and usefulness of the response may not be as good as a smaller context window.

The best size will of course, depend on your use case.