The model loads first. Then type a line, write one, or pick a sample.
The line, and where the model looks for the selected word piece What it wrote, one word piece at a time · click a pieceRead a line on the first tab, then click on it here.
The line is shown at its real proportions. The encoder squashed it into 384 × 384 and cut that into a 24 × 24 grid, so on the real line each patch is a tall, thin strip: 1/24 of the width and 1/24 of the height.
Each map is one attention head: where the selected patch gathers information from, drawn at the line's proportions. Heads specialise: some look only at the patch itself or its neighbours, many gather the whole column it sits in (the same stretch of the line, from top to bottom), and some spread over all the ink. Click one to show it large.
The diagram lights up block by block while this panel shows what comes out of each stage.
The same word, “bank”, in 8 sentences. It is the 6th word piece in every one, so it starts with exactly the same numbers. The real TrOCR reads each line; we take the 256 numbers it holds for “bank” and compare every pair with cosine similarity (1.00 = pointing the same way, 0 = unrelated).
In attention, every piece makes three vectors from its current state: a query (what it is looking for), a key (what it offers, for other pieces' queries to match) and a value (what it hands over when it is matched). A query is compared with every key; the best matches decide how much of each value is mixed in. Here the real TrOCR decoder reads each phrase. The target piece is always the second one, so its position is the same and only the word before it differs.
Each strip is the 256 numbers of the vector in the selected layer, as 8 heads × 32 numbers (orange positive, blue negative). Top: the vectors themselves, which look almost the same. Bottom: each minus the average over all contexts, which shows exactly what the context changed.
CTC is forced to move left to right. TrOCR is free to look anywhere, yet a left-to-right diagonal appears on its own: each piece looks at the next stretch of the line. This is the line from the first tab, cross-attention summed over the image's height.
Attention can lose its place in ways CTC cannot: it can skip or repeat words, or invent plausible text, and it squashes every line into the same 384 pixels. Run the real model on these and see what happens.