Definition: neural network

The major AI architecture. A neural network is employed for many pattern recognition applications; however, its most popular use is the creation of language models used by ChatGPT, Gemini and other chatbots. Loosely based on the human nervous system, a neural network is technically an "artificial" neural network (ANN).

The neural network is used in image, language and speech recognition, text-to-speech conversion, robotics, diagnosing, forecasting and generative AI. Unlike regular applications that are programmed for precise results (if-then-else), neural network models are "trained" and fine-tuned using millions, billions, even trillions of examples of text and images. See AI secret sauce, AI programming, AI training, AI model and generative AI.

Layers and Nodes
Neural networks comprise one input and one output layer and any number of hidden layers in between (see below). Each layer contains a number of nodes, called "neurons," which are the primary data elements in a neural network architecture. The neuron's mathematical values are computed with all the neurons in the next layer.

Datacenters can have thousands of servers, each containing thousands of neurons. The more servers, the larger the neural network and the more comprehensive the training. For smaller AI applications, a single desktop machine can contain a neural network; for example, see DGX Spark.

GPUs and Servers
AI servers contain four to 16 GPUs, each with their own processor and memory; for example, NVIDIA's H100 contains eight GPUs (see H100). In a large datacenter, there can be tens of thousands of servers, and it can take weeks and months to train huge language models. To reduce the training time of ever-increasing neural networks to days instead of months, it is estimated that a million or more GPUs may be required. See GPU and AI training vs. inference.

A Neuron in a Neural Network

Nothing like "if-then-else" business logic, neural networks cannot be debugged or reverse engineered like regular computer programs. This is a single neuron. See AI weights and biases.

Tracing a Neuron
The neural network is a pattern detection system. Sentences are turned into tokens that are parts of words, and then vectors, which are one-dimensional arrays. The vector arrays become a matrix that contains the neural network weights.

Millions of neurons process each token to predict the next one, and training language models means constantly predicting what comes next.

The World's Information = Giant Matrix
Essentially, AI companies have taken nearly all the information ever published, which is online and available over the Internet and turned it into the world's largest mathematical matrix. See AI secret sauce.

There Are Many Network Designs
The following diagrams from the Asimov Institute in the Netherlands reveal the variety of neural network architectures that have been created. For a neural network example that recognizes the letters of the alphabet, which is easier to grasp than language recognition, see convolutional neural network.

Neural Network Architectures

AI networks are one of the most researched areas of computing in the 21st century. The examples above from the Asimov Institute in the Netherlands reveal the variety of network architectures that have been created. (Images courtesy of Fjodor van Veen and Stefan Leijnen (2019). The Neural Network Zoo.)