Self-Attention in Transformers

Medium

neural-networks Google OpenAI Anthropic

Multiple Choice

It measures similarity between tokens; scaling by sqrt(d_k) prevents the softmax from producing near-one-hot distributions with vanishing gradients It measures the Euclidean distance between tokens; scaling converts it to a cosine similarity It computes the cross-correlation of tokens; scaling normalizes for variable sequence length It computes the mutual information between tokens; scaling adjusts for the number of attention heads

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations