Layers of weighted sums passed through a non-linear function, with the weights fitted by gradient descent. The base structure under every current AI model.
Neural Network is the base structure under every current AI model: layers of weighted sums passed through a non-linear function, with the weights fitted by gradient descent. A layer multiplies its input by a weight matrix, adds a bias, and applies an activation such as ReLU or GELU. Stacking layers is what makes the whole function expressive enough to fit language, images or audio.
Without an activation function, stacked linear layers collapse into one linear layer and depth buys nothing. The non-linearity is what lets a deep network approximate arbitrary continuous functions, and it is why the manifold hypothesis is usable in practice: the network bends the input space until the structure that matters becomes separable.
The weights and biases are the parameters, and their count is the figure quoted in a model's name. They start random and are updated by backpropagation against a loss function. Once training stops they are frozen into a checkpoint, and everything the model knows is held in them as parametric memory.
