A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Neural networks compose non-linear functions (neurons) across layers. Each neuron computes a weighted sum plus bias through an activation function. Universal approximation theorem states sufficiently wide networks approximate any continuous function.
You have already worked with linear regression, which fits a single linear function — a weighted sum of inputs — to data. Neural networks extend this idea dramatically: instead of one linear transformation, they stack many layers of simple units (neurons), each with its own weights, and crucially, each followed by a non-linear transformation. This stacking of non-linearities is what makes neural networks capable of approximating extraordinarily complex functions.
A single neuron does something familiar: it computes a weighted sum of its inputs plus a bias, exactly like linear regression. For inputs x₁, x₂, ..., xₙ with weights w₁, w₂, ..., wₙ and bias b, the pre-activation value is z = w₁x₁ + ... + wₙxₙ + b. So far this is just linear regression. The key addition is the activation function f, applied to z: the neuron's output is f(z). Common choices are ReLU (f(z) = max(0, z)), sigmoid (f(z) = 1/(1 + e-z)), and tanh. These functions are non-linear, and that non-linearity is everything.
Why does non-linearity matter so much? Because the composition of linear functions is always linear. If you stacked ten layers with no activation functions, the whole network would be equivalent to a single matrix multiplication — as limited as linear regression. Once you introduce non-linear activations, the network can bend, curve, and fold the input space in ways a single linear function cannot. The universal approximation theorem makes this precise: a sufficiently wide single hidden layer can approximate any continuous function on a compact domain to arbitrary accuracy. Deeper networks with more layers can approximate the same functions with fewer neurons — this is why depth is valuable.
Networks are organized into layers: an input layer that receives the raw features, one or more hidden layers where the actual computation and representation learning happens, and an output layer that produces the prediction. Each layer transforms its inputs into a new representation, and as you go deeper, these representations become increasingly abstract. In a network trained on images, early layers might detect edges, later layers shapes, and the final layers might represent concepts like "cat" or "chair."
Training a neural network means finding the weight values that minimize prediction error on training data. This is done by computing the gradient of the loss with respect to every weight — which requires calculus, specifically the chain rule applied repeatedly through the layers (the backpropagation algorithm). This is why derivatives and partial derivatives are prerequisites: without the ability to compute how the loss changes with respect to each weight, you cannot update the weights in the right direction. The connection to linear regression is also direct: a neural network with no hidden layers and a linear activation is exactly linear regression.