Stanford University School of Engineering
77 minutes
Word Window Classification and Neural Networks in Natural Language Processing
This lecture develops the theory of word-window classification and feedforward neural networks in NLP: given a fixed-size context window around a center word, word vectors are concatenated into a single input vector and fed either into a softmax classifier (with cross-entropy loss, formally derived as a special case of the general cross-entropy between a one-hot target distribution and a predicted distribution) or into a neural network with a hidden layer of sigmoid units before a final scoring layer, trained with a max-margin (hinge) loss that separates scores of "true" versus "corrupt" windows by a margin. It formalizes backpropagation as repeated application of the chain rule with reusable intermediate error terms ("delta"), and establishes when word vectors should be kept fixed versus fine-tuned as part of training, situating these mechanisms within representation learning and neural network theory in computational linguistics.