Credit: Dam Minh Tien

M1: Fundamentals of Language Understanding

In this module, you will learn about one-hot encoding, bag-of-words, embeddings, and embedding bags. You will also gain knowledge of neural networks and their hyperparameters, cross-entropy loss, and optimization. You will then delve into the concept of language modeling with n-grams. The module also includes hands-on labs on document classification with PyTorch and building a simple language model with a neural network.

Learning Objectives

• Explain how to use one-hot encoding, bag-of-words, embedding, and embedding bags to convert words to features.
• Describe how neural networks are used for document categorization prediction.
• Describe the process of training and optimizing neural networks for document categorization.
• Describe the applications of N-Gram language models in natural language processing.
• Implement document classification using torchtext and PyTorch.
• Build and train a simple language model with a neural network.

1. Language Understanding with Neural Networks

1.1. Converting Words to Features

• One-Hot Encoding: This method converts categorical data into feature vectors. Each word is represented as a vector where only one element is '1' (indicating the presence of that word) and all others are '0'.
• Bag of Words: This representation aggregates one-hot encoded vectors to depict an entire document. It captures the presence and frequency of words.

• Embeddings and Embedding Bags:
  • Embeddings replace one-hot vectors with lower-dimensional representations, simplifying computations.
  • Embedding Bags sum the embeddings of words in a document, providing an efficient way to represent the document.
• Using PyTorch: The lecture also covered how to implement these concepts in PyTorch, including initializing embedding layers and using embedding bags to process text data.