Artificial neural networks originate as abstractions of biological neurons treated purely as computing elements, beginning with the McCulloch-Pitts unit — an unweighted summation of excitatory inputs against a threshold, vetoed by any active inhibitory input — and generalizing to the perceptron, which replaces the plain sum with a weighted linear combination plus a bias absorbed as an extra unit input. The perceptron's threshold nonlinearity is non-differentiable, so a linear-output variant (the adaptive linear unit) is trained instead by stochastic gradient descent on squared error, which differs from the perceptron rule's objective of minimizing the distance of misclassified points to the separating hyperplane. Because a single threshold unit can only realize linearly separable functions, the central theoretical result is that composing such units in layers — equivalently, having earlier units construct a new feature basis — makes non-linearly-separable functions computable, which is the foundational argument for depth in machine learning.
Artificial neural networks originate as abstractions of biological neurons treated purely as computing elements, beginning with the McCulloch-Pitts unit — an unweighted summation of excitatory inputs…