Worked attention computation with real small numbers: three tokens with given query, key and value vectors of length four, showing dot products, scaling by the square root of dimension, softmax weights, and the final blended output vector, with a toggle between two query vectors.
The cast, produced by the learned matrices
q for "sat"
"the"k1100 v1002
"cat"k0122 v0210
"sat"k2010 v2011
"sat" is the last token, so all three are visible to it. Amber = query, teal = keys, plain = values.
Step 1 · similarity: dot q with each key
Step 2 · scale by √d = √4 = 2
Longer vectors make bigger dot products by accident; dividing by √d keeps softmax from saturating.
Step 3 · softmax the scores into weights
the0.06
cat0.69
sat0.25
Step 4 · blend the value vectors with those weights