Definition: backpropagation

Short for "backward propagation of errors," it is the primary method used to train an AI language model. A model's neural network learns patterns in training data by repeatedly predicting the next token (word or part of a word) and adjusting its numerical parameters in the backward pass.

The Model Does Not Contain the Data
A neural network model is not a database of facts and figures. It does not contain actual data. It contains billions of learned numerical values that encode patterns and relationships of data. To be sure, the neural network is the most peculiar data processing activity that has ever come down the pike!

Step 1 - Forward and Compare
Training starts by converting a batch of text into tokens with a numerical token ID and an embedding table initialized with random numbers for each token (see example below). The forward pass predicts the next token, and the predictions are compared with the actual next token. The difference between the predicted and actual next token is known as the "loss function," and the "cost function" is the average loss for the batch of training examples.

Step 2 - Backward and Adjust
Using this error information, backpropagation goes back through the network and with the use of a "gradient descent" algorithm increases or decreases the neuron's weights and biases and the embedding table to generate a better result the next time around. A backpropagation pass generally follows a forward pass.

Prior to backpropagation, training was extremely slow, and backpropagation algorithms are considered one of the major breakthroughs in training AI models. AI scientists designing backpropagation algorithms require a strong calculus background. See weights and biases.

Quadrillions and Quintillions of Changes
The forward and backward passes are repeated over and over until the model achieves the desired outcome. Very large language models that have billions of tokens can perform the forward pass and backpropagation steps millions of times, which means the total number of changes to the nodes and embedding table can be in the quadrillions (see space/time).

Adjust the Weights and Biases

The backpropagation algorithms constantly adjust the mathematical values between the neurons. In a large language model, there can be billions and trillions of neurons. See neural network.

An Embedding Table

This embedding table shows only three hidden layers. However, large language models can have hundreds of hidden layers and hundreds of neurons in each layer. In the forward pass, the neuron uses the token ID to retrieve the embedding vectors that it multiples with its own weights. It then applies an activation function to produce a single number that are the values that pass through the neural network. See AI token.