Self-Attention in Transformers

Medium

neural-networks Google OpenAI Anthropic

In the transformer architecture, self-attention computes query, key, and value vectors. What does the dot product of queries and keys represent, and why is it scaled?

Multiple Choice

It measures similarity between tokens; scaling by sqrt(d_k) prevents the softmax from producing near-one-hot distributions with vanishing gradients It measures the Euclidean distance between tokens; scaling converts it to a cosine similarity It computes the cross-correlation of tokens; scaling normalizes for variable sequence length It computes the mutual information between tokens; scaling adjusts for the number of attention heads

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Self-Attention in Transformers

Multiple Choice

Related problems

Deep-dive guides