Machine learning with many stacked layers of learned transformations, trained end to end by gradient descent, so features are learned from data rather than designed by hand.
Deep Learning is machine learning with many stacked layers of learned transformations, trained end to end by gradient descent. The depth is the point: each layer builds a representation from the one below it, so the features that earlier approaches needed an engineer to design are learned from the data instead.
A forward pass runs the input through every layer to produce an output and a loss. Backpropagation applies the chain rule backwards through the network to get the gradient of that loss with respect to every parameter, and an optimiser such as Adam takes a small step against the gradient. That repeats over batches for one or more epochs, writing checkpoints along the way.
Three things arrived together: data at internet scale, GPUs that make the underlying matrix multiplications cheap, and architectures that train stably at depth, notably residual connections and the transformer. Increasing any of parameters, data or compute improves loss along a predictable curve, which is the finding that justified current model sizes and the pre-training runs behind them.
