hidden states - latent superpositions of concepts (each represented as a direction in a high-dimensional space) - as are their outputs. Each FFN - which there can be dozens to hundreds - functions as a detector-generator: they detect concepts in the input, and encode the concepts that result from...

Show Post

... this combination of concepts into the output. Between each FFN is a multiheaded attention block. Each FFN codes in clues about what sort of information from the input will be of use to the next layer, given the context at hand; the attention mechanism seeks said relevant information from...

Show Post

... the context, and merges said latent concepts into the subsequent FFN (add + norm rather than replacement). As you approach the final FFN layers, the main work of the FFNs switches from reasoning toward converging to an output direction. The final hidden state however now has to face the...

Show Post

... challenge of going from a superposition of concepts in a high dimensional space, down to a low-dimensional linguistic space. Sometimes, it will be very close to a token position - e.g. if it's writing "The tomb of Tutankh...", the output will be pretty much right on top of "amun", because...

Show Post

... there's only really one valid linguistic path. By contrast, if it's "The hungry boy climbed the mango tree and...", there's dozens of ways you could go to talk about him picking a mango; the final latent state will land between the positions of different tokens that could lead to expressing...

Show Post