AI Term of the Moment

AI ethicist


Look Up Another Term


Definition: hyperparameters


The design choices and settings of a neural network model. Hyperparameters are chosen by the designer of the model, who may adjust them throughout the training and inference stages. See neural network.

Training Hyperparameters
Examples of training hyperparameters are the number of hidden layers in the network, the number of nodes/neurons per layer and the training epochs (number of passes through the dataset). The number of training samples in the dataset and learning rate (how much the weights are changed) are also hyperparameters.

Inference Hyperparameters
Settings may also be changed for executing the model (inference stage). For example, "temperature" controls the randomness and sharpness of the answer. Token minimums and maximums (min_tokens, max_tokens) determine the length of the generated answer, and a frequency penalty reduces repetitive wording. Forbidden words (bad_words) ensures profane language is not used.

Parameters Are the Weights and Biases
In contrast to hyperparameters that are adjusted and modified by human engineers, the parameters are the numeric weights and bias values that are set at the design stage but constantly adjusted throughout the training passes (see backpropagation). See weights and biases and training vs. inference.

Although a neural network can be created with just a handful of trainable parameters, small language models (SLMs) have from a few thousand to a couple hundred thousand parameters, while large language models (LLMs) can have billions and trillions of parameters. In 2026, Chinese AI lab Moonshot introduced Kimi K3, an open source model with 2.8 trillion parameters.