The latest approach to training AI models and generating answers (inference). Attention is a mathematical scoring system that enables the model to learn language patterns more effectively. It computes a "relationship score" between every word in the text to learn which words depend on each other. See
neural network and
GPT.
Training
In the training stage, Attention creates a multi-dimensional matrix (a grid) of every word it reads. The GPU computes scores for every intersection in this "Attention Matrix," which provides a mathematical relationship between every word in the block of text.
Training involves many forward passes (there can be millions and billions), and with each forward pass, the next word is guessed. If the guess is incorrect, an error score (loss) is computed, and the backward pass (backpropagation) adjusts the weights (values) in the connections between the neurons in the neural network layers.
Inference
The inference software, which generates answers for the user, must create complete sentences that make sense. The first step is reading the user's prompt. This "prefill" stage uses Attention math to understand the context of the user's prompt by first storing it in a memory cache known as the KV Cache.
It then generates each word in the answer by running through a three-part pipeline. Part 1 passes the data in the cache through the model's layers, which outputs a massive list of every word it knows along with a percentage score of how likely it is the word that comes next. Part 2 executes an algorithm that chooses the next word and the memory cache is updated. Part 3 repeats the loop until the answer is completed.
Every Token Generated Requires a Forward Pass
Every token in the answer that the inference software generates means a forward pass through all the layers in the model. If a model runs at 100 tokens per second, the inference engine is executing 100 forward passes through the neural network in one second.
Not Always!
In an AI data center, when thousands of people are online simultaneously, inference passes use a lot of machine time. As a result, AI companies often use smaller "draft models" that guess multiple output tokens in one pass. However, because users often pay per output token, the machine time savings are not always passed on. Nevertheless, over time, the more effective the techniques for reducing passes through the model, AI companies can lower prices to be more competitive.