A feed-forward neural network is a stack of alternating linear maps and non-linear squashes. Each layer multiplies its input vector by a weight matrix, adds a bias vector, and applies an activation function elementwise. The forward pass is nothing but that, repeated.
Worked example: a forward pass by hand
Work one all the way through, because a single numeric example dissolves most of the mystique. With inputs (0.60, −0.40), a hidden weight matrix [[1.2, −0.8], [0.5, 0.9]], hidden biases (0.10, −0.20), output weights (1.5, −1.1) and an output bias of 0.30, the hidden pre-activations are 1.14 and −0.26. A ReLU activation, max(0, z), passes the first through unchanged and sets the second to zero, so the second hidden unit contributes nothing to this particular input. The output pre-activation is 1.5(1.14) − 1.1(0) + 0.30 = 2.01, and a sigmoid turns that into a probability of 0.8818.
The activation is what makes the network worth building. Replace ReLU with the identity and the same weights give 0.9085, but more importantly the whole network collapses into a single linear map with effective weights (1.25, −2.19) and an intercept of 0.67. A stack of linear layers is a linear layer, so without a non-linearity depth buys nothing at all.
Advertisement
How is a neural network trained, and when does it help?
Backpropagation, set out by Rumelhart, Hinton and Williams in 1986, is how the weights get fitted. It computes the derivative of the loss with respect to every weight by applying the chain rule backwards through the network, reusing the intermediate results the forward pass already produced, and gradient descent then steps every weight a little way downhill. The count of parameters is what should worry an analyst: a modest network with two hidden layers of fifty units on twenty inputs carries about 3,600 weights, which is more free parameters than most financial panels have observations.
The hand computation of a backward pass, the training loop, and everything about sequence models and language belongs to the deep-learning node in the quant-code track. What belongs here is the vocabulary and one honest verdict: on structured, tabular, low-signal financial data, gradient-boosted trees usually beat neural networks, and the cases where networks win involve text, images or genuinely large sequence data.