Context Window Limitations

Medium

llms Anthropic OpenAI Google

What is the primary technical challenge that limits the context window size of transformer-based LLMs?

Multiple Choice

The tokenizer cannot encode sequences beyond a fixed length Self-attention's quadratic time and memory complexity with respect to sequence length The positional encoding scheme only supports a fixed number of positions GPU memory is limited to storing a fixed number of token embeddings

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Context Window Limitations

Multiple Choice

Related problems

Deep-dive guides