Language Models (LM)

What Is a Language Model?

• A model that learns a probability distribution over sequences of text tokens.
  • Model
    • An abstraction of patterns found in real-world data.
      • Which patterns does a language model represent?
        • Probability Distributions
          • The chain rule gives P(I, like, cats) = P(I) × P(like | I) × P(cats | I, like). With example probabilities 0.5, 0.8, and 0.6, the joint probability is 0.24.
        • Text Sequences
          • An ordered sequence of tokens.
          • Text can be segmented into tokens.

Tokens and Vocabulary

• Token: the basic unit processed by a language model.
  • Why use tokens rather than individual characters or entire words?
    • Subword tokens offer a practical balance between vocabulary size and coverage.
    • Subtopic 2
  • Vocabulary: the set of tokens recognized by a tokenizer.

Types of Language Models

• Masked Language Models
  • BERT
  • Reconstruct masked tokens using surrounding context.
  • Typically uses bidirectional context.
  • Example
    • Original text: “I enjoy learning programming.” / Masked training input: “I enjoy [MASK] programming.” / Expected token: “learning”.
• Autoregressive Language Models
  • GPT (a representative family)
  • Predict the next token from the preceding tokens.
  • Uses a causal, left-to-right context.
  • Example
    • Original text: “I enjoy learning programming.” / Autoregressive input: “I enjoy learning” / Next-token target: “programming”.