Definition: neural network

The architectecture of today's language models, which provides the source of learned knowledge that chatbots such as ChatGPT, Gemini and Claude use to answer questions. A neural network is essentially a "pattern recognition" system. Very loosely based on the human nervous system, a neural network is technically an "artificial" neural network (ANN). Hypothesized in the 1940s, there were elementary neural networks as far back as the 1950s.

The neural network is used in image, language and speech recognition, text-to-speech conversion, robotics, diagnosing, forecasting and generative AI. Unlike regular applications that are programmed for precise results (if-then-else), large language models are "trained" on millions and billions of words and images, and the answers generated for users are provided by "inference" processing (see AI inference). See AI secret sauce, AI programming, AI training, AI model and generative AI.

Servers and GPUs
Datacenters can have thousands of servers. The more servers, the larger the neural network and the more comprehensive the training. For smaller AI applications, a single desktop machine can contain a neural network; for example, see DGX Spark.

AI servers contain a number of GPUs, each with its own processor and memory; for example, NVIDIA's H100 contains eight GPUs (see H100). In a large datacenter, even with tens of thousands of servers, it can take weeks and months to train huge language models. To support the neural networks of ever-larger models as well as reduce training time from months to days, it is estimated that a million or more GPUs may be required. See GPU and AI training vs. inference.

Layers and Nodes

A neural network is a "math machine" that learns from examples. It comprises multiple layers of computational units called "nodes" or "neurons" that are mathematically connected to each other (see diagrams below). There is an input layer, an output layer and any number of "hidden" layers in between.

A Single Neuron/Node

A neuron, also called a "node," is a mathematical processing unit within the neural network, and a single GPU processes many neurons simultaneously. The activation function determines if the value leaving the neuron should be altered. The weights and bias values are modified in the training passes when predictions are compared against the correct answers. If this all sounds very strange... it is! See AI weights and biases.

Tracing a Neuron
The neural network is a pattern detection system. The text in sentences (word, part of a word, etc.) is turned into a token that is converted into a one-dimensinal array called a "vector" or "embedding." Each layer transforms the embeddings mathematically by looking at the tokens in the sentence and making the embeddings more meaningful. The last layer predicts the next token.

Training language models means constantly predicting what comes next and evaluating that prediction for accuracy (see AI training passes). Essentially, AI companies have taken all the information ever published online and turned it into a giant maze of numbers! See AI secret sauce and copyright infringement.

A Tiny Network

This neural network example has only 17 neurons/nodes in three hidden layers and might be used for limited pattern recognition in a small sensor.

A Larger Network

This neural network example has 8,192 neurons/nodes in eight hidden layers for a total of 65 thousand nodes and 4.3 billion connections that are adjusted in the training stages. Now imagine a neural network with 500 billion nodes and you can begin to visualize the complexity of large AI models.

There Are Many Network Designs
The following diagrams from the Asimov Institute in the Netherlands reveal the variety of neural network architectures. Note that whereas only a few layers are shown in each example, there can be hundreds or even thousands of layers depending on the network type.

The CNN: Easier to Understand
Convolutional neural networks (CNNs) are used to recognize images, text and handwriting, and their operation is easier to understand than chatbot models that appear to know everything. For details, see convolutional neural network.

Neural Network Architectures

AI networks are one of the most researched areas of computing in the 21st century. These examples are a sample of the designs. There are many more. (Images courtesy of Fjodor van Veen and Stefan Leijnen (2019). The Neural Network Zoo.)