Positional Encoding in Transformers

Medium

neural-networks Google OpenAI DeepMind

Why do transformers require positional encoding, and what property makes sinusoidal positional encodings effective?

Multiple Choice

Self-attention is permutation-invariant; sinusoidal encodings let the model learn relative positions because the encoding of position p+k can be represented as a linear function of the encoding at position p Self-attention only processes pairs of tokens; positional encoding adds sequential context from neighboring tokens Transformers process tokens in reverse order; positional encoding corrects this by indicating the forward sequence Positional encoding reduces the sequence length to a fixed size so attention can operate in constant time

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Positional Encoding in Transformers

Multiple Choice

Related problems

Deep-dive guides