(1) For crypto, see
AI crypto token.
(2) An AI token is a mathematical representation of text. When text is fed into an AI model for training, it is first turned into tokens. When a prompt is given to a chatbot to generate an answer, the prompt is turned into tokens, and when the inference software combined with the chatbot software generates an answer, the output tokens are turned into text so people can read it. To learn how tokens are analyzed in the training phase, see
backpropagation.
Text to Tokens to Embedding Vectors
"Tokenizer" software turns text into tokens, which are numbers that represent a word, fragment of a word, punctuation (, . !) or symbol ($ # -). Each token is turned into an array of numbers known as an "embedding vector." The vectors start out as random numbers but each training pass adjusts them to become more relevant. To deliver an answer, the detokenizer turns the generated tokens back into text for human consumption. See
AI transformer and
vector database.
Image Tokens
Images may be turned into tokens as well and the tokens represent visual data. For example, an image may be broken into "patches" of 16x16 pixels, and each patch is converted into a numerical representation.
Example: "This is a token."
Using the GPT model as the tokenizer, the sentence is first broken into five tokens and then mapped to embedding vectors in the "embedding table" (see below), which can contain millions or billions of numbers. There is an embedding vector for each token in the model's vocabulary.
As the model is trained, the node's weights and biases along with the vectors in the embedding table are repeatedly updated. In inference, the embedding table values remain fixed for reading.
Following is a brief sample of vectors in an embedding table. In a real model, there can be hundreds and thousands of embedding vectors in order to represent the complex relationships in human languages.
Sentence: "This is a token."
Word Token ID Embedding Vector
"This" 1052 [0.25 -0.73 0.91 0.14 -0.52]
" is" 318 [0.61 0.22 -0.45 0.83 0.17]
" a" 257 [-0.34 0.76 0.12 0.59 0.41]
" token" 11241 [0.88 -0.15 0.63 0.29 -0.72]
"." 13 [0.19 0.42 -0.81 0.36 0.55]