Multi-Head Attention

Medium

neural-networks Google OpenAI Anthropic

Why does the transformer use multi-head attention instead of a single attention mechanism with the same total dimensionality?

Multiple Choice

Multiple heads let the model jointly attend to information from different representation subspaces at different positions Multiple heads reduce the total number of parameters compared to a single large attention layer Multiple heads ensure that attention weights always sum to exactly 1 across all tokens Multiple heads are needed to make attention compatible with residual connections

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Multi-Head Attention

Multiple Choice

Related problems

Deep-dive guides