Multi-Head Attention

Medium

neural-networks Google OpenAI Anthropic

Multiple Choice

Multiple heads let the model jointly attend to information from different representation subspaces at different positions Multiple heads reduce the total number of parameters compared to a single large attention layer Multiple heads ensure that attention weights always sum to exactly 1 across all tokens Multiple heads are needed to make attention compatible with residual connections

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations