Large Language Models (LLM)

What Is an LLM?

A language model trained at a significantly larger scale
- Size
  - Training Data
  - Parameter Count
  - Training Compute
- Why Language Models Scale
  - Other algorithms
    - Object Detection
      - Representative model: YOLO
    - Topic Modeling
      - Representative model: LDA
  - Self-Supervised Learning
    - Does not require manually labeled data
    - By contrast, supervised learning relies on labeled examples

How to Build an LLM

Prepare Training Data
- Common Sources of LLM Pretraining Data
  - Web Data
    - │ ├── Common Crawl
      - https://commoncrawl.org/?utm_source=chatgpt.com
      - https://commoncrawl.org/get-started?utm_source=chatgpt.com
    - Custom-built web crawlers
    - {\cHFFFFFF}{\cH00FF00} {\cHFFFFFF}{\cH00FFFF} {\cHFFFFFF}{\cH00FFFF} {\cHFFFFFF}{\cH00FFFF}{\cH00FFFF}
  - 1. Professional data
    - Ideas - Academic Papers
    - Books
    - Code repositories
  - Licensed Data
    - │ Data obtained through authorization agreements
      - Pirate
        
        Proven case: Anthropic / In 2025, the U.S. Federal Court of California heard the Bartz v. Anthropic case. / Court documents disclose that Anthropic obtained: / Source of data / Books 3 about 197,000 copies by 2021 Load / Library Genesis (LibGen) Load / Pirate Library Mirror (PiLiMi) Load
  - Synthetic Data
    - └ - Data generated by models or programs
    - If there’s an AI-generated data, why is the data missing?
      - Number
        
        AI can generate text on a continuous basis, theoretically close to infinite.
      - Quality
        
        AI does not necessarily produce the correct content or have sufficient information value.
        
        Model collapse: The use of data generated by previous generations for training on re-entry indiscriminately may lead to the gradual loss of long tailings from the distribution of raw data.
      - Diversity
        
        Even if the generated content is correct, the lack of new knowledge and diversity may prevent the continued upgrading of modelling capabilities.
        
        Innovation?
        
        Just like humans: summarizing leads you to the pattern, backs up because you’re going to explain, you’re going to extrapolate a testable prediction, you’re going to test the hypothesis. / These three reasonings complement each other and constitute a more complete process of scientific research. / Stanford Philosophy Encyclopedia. / There is also a key distinction: the development of a hypothesis that others have not made is a potential innovation; it can be further recognized as an innovation only if it is validated and new value or knowledge is generated.
        
        https://chatgpt.com/c/example
Model Architecture
- Architecture
  - Transformer
    - Based on seq2seq
      - What It Is
        
        It is a modelling method for converting input sequences into output sequences.
      - Modules
        
        Encoder
        
        Encoding input sequences as internal expressions
        
        It’s kind of like compression.
        
        Enter token, output is the final hidden state of the input
        
        Process
        
        Step 1: Tokenizer, convert text to Token ID
        
        Step 2: Embeding, convert Token ID to vector
        
        Step 3: RNN Encoder calculates the hidden status in order
        
        Legend
        
        Enter Text / “I love cat” / ↓ / Tokenizer / [1, 2, 3] / ↓ / Embedding / x₁ = [1,0] / x₂ = [0,1] / x₃ = [1,1] / ↓ / RNN Encoder / h₁ = RNN(x₁,h₀) / h₂ = RNN(x₂,h₁) / h₃ = RNN(x₃,h₂) / ↓ / Output hidden status associated with context / h₃ = [0.828,0.881] / ↓ / Pass to Decoder
        
        Codec
        
        Based on internal indications, the output sequence is gradually generated.
        
        Process
        
        (untitled topic)
        
        Just like the encoder, there’s a lot of steps to turn the hidden state into a probability.
        
        It’s called hiding. It means in the middle. / state := []float64{0, 0} / for _, token := range tokens { / state = RNN(token, state) / }
        
        Linear
        
        Linear: Convert hidden status to the fraction of each candidate word
        
        logit scores
        
        https://chatgpt.com/c/example
        
        Softmax
        
        Convert logit scores to probability
        
        Legend
        
        Encoders and decoders use RNN.
        
        Circulatory neural network.
        
        Gradient
        
        Gradient is the attribute of the loss function to a variable and is used to measure how changes in the variable affect the loss.
        
        Problems
        
        Gradient Missing
        
        Gradient explosion
        
        The cycle here means that the nerve network, when dealing with sequences, transmits the status of the previous step to the next.
      - Problems
        
        Solve
        
        Attention Mechanism
        
        It inherited the classic Seq2Seq encoder-coder idea, but replaced the traditional RNN loop calculation with Attention.
        
        Serial Calculation
        
        RNN Encoder must rely on the previous Token’s hidden state, making it difficult to calculate in parallel within the sequence.
        
        Information Bottleneck
        
        Classic Seq2Seq compresses the entire input sequence into a fixed-length vector from which Decoder can hardly recover all important information.
        
        no-loss compression by analogy
        
        A fixed-volume vector is unlikely to represent all input sequences of any length without loss.
        
        The traditional RNN Seq2Seq uses a fixed long vector, essentially a trade-off between information capacity and costing.
        
        Traditional RNN Seq2Seq / │ / Ideas - Fixed Length / Idea - Less cost to calculate and store / └ - Long series information bottlenecks / ↓ / RNN + Attention / │ / Ideas - Keep Hidden Multiple Locations / Ideas-Decoder Read Information on Needs / └ Increased costs of calculation and storage / ↓ / Transformer / │ / Ideas - Attention becomes the core computing mechanism / Ideas - no longer relying on RNN serial transmission / Ideas - Improved training parallel capabilities / └ - Face long-sequence costing problems
      - Process
        
        Training process: how models learn to translate
        
        Encoder + Decoder + Los + Reverse
        
        Enter Sequence (English) Output Sequence (Chinese) / I love cats.
        
        Training data: x = 2, y = 10 / │ / ▼ / 1 Forward Dissemination / ŷ = ax²+b / a=1，b=1 / ŷ=5 / │ / ▼ / 2 Calculating Los / L=(5-10)²=25 / │ / ▼ / 3 Reverse transmission / ∂L/∂a=-40 / ∂L/∂b=-10 / │ / ▼ / 4 Optimizer update parameters / a = 1.4 / b = 1.1 / │ / ▼ / 5 Forward transmission again / ŷ=1.4×4+1.1=6.7 / │ / ▼ / 6 Calculating New Los / L=(6.7-10)²=10.89 / │ / ▼ / Repeated training
        
        https://chatgpt.com/c/example
        
        Worked Gradient-Descent Example: ŷ=ax²+b. Treat a and b as trainable parameters. This is a simple parameterized mathematical function—not a multilayer neural network—but the same training principles apply. / Define a training sample: input x=2, target y=10, and initial parameters a=1, b=1. The prediction ŷ has not yet been computed. / Step 1 — forward pass: ŷ=ax²+b=1×2²+1=5; the predicted value 5 differs from the target 10. / Step 2 — loss: use squared error L=(ŷ−y)², appropriate for this regression example rather than token classification. Thus L=(5−10)²=25. Loss calculation alone neither differentiates nor updates parameters. / Step 3 — gradients via the chain rule: since L=(ax²+b−y)², the partial derivatives are ∂L/∂a=2(ŷ−y)x² and ∂L/∂b=2(ŷ−y). At the current values, ∂L/∂a=2(5−10)×2²=−40 and ∂L/∂b=2(5−10)=−10. The other parameter is held constant for each partial derivative. These gradients indicate local changes in loss, not the final best parameter values. / Step 4 — optimizer: choose learning rate η=0.01 and apply θ_new=θ_old−η∂L/∂θ. Then a_new=1−0.01×(−40)=1.4 and b_new=1−0.01×(−10)=1.1. The model is now ŷ=1.4x²+1.1; its mathematical form has not changed, only its parameters. / Step 5 — evaluate the updated model on the same sample: ŷ=1.4×2²+1.1=6.7 and L=(6.7−10)²=10.89. One update reduced loss from 25 to 10.89, but this is not guaranteed for every learning rate or update; excessively large steps can increase loss. / The essential sequence is forward pass → loss calculation → gradient computation → parameter update → new forward pass. For classification or next-token prediction, a different loss (commonly cross-entropy) is more appropriate.
        
        Seq2Seq Training: from Input Sequence to Output Sequence. / What it is: Sequence-to-sequence learning maps an input sequence to an output sequence. A classic implementation is an RNN encoder–decoder. Training adjusts parameters to increase the probability of the reference output; the loop is forward pass → loss → backpropagation → parameter update. / 1. Prepare data: source = I love cat; target = 我爱猫 <EOS>. During training, both source and target are provided by the dataset. / 2. Encode: tokenize I love cat into IDs such as [1, 2, 3], then look up trainable embeddings x₁, x₂, x₃. The RNN encoder computes h₁ = RNN(x₁,h₀), h₂ = RNN(x₂,h₁), and h₃ = RNN(x₃,h₂). The final state h₃ summarizes the source (with possible information loss) and initializes the decoder. / 3. Decode: start from <BOS>. At each step, the RNN uses the previous target token embedding and hidden state, a linear layer produces vocabulary logits, and softmax gives a distribution over candidate tokens. Example probabilities for the correct next tokens are P(我|X,BOS)=0.6, P(爱|X,我)=0.7, P(猫|X,我爱)=0.8, and P(EOS|X,我爱猫)=0.9, where X=I love cat. / Teacher forcing: during training the decoder receives the previous ground-truth target token, even when the model predicted incorrectly; it does not need to pick the maximum-probability token. There is no rule requiring a prediction probability above 50%. / 4. Calculate loss: token prediction generally uses cross-entropy. Each position contributes Lₜ = -ln P(correct token), so total example loss is L = -ln(0.6)-ln(0.7)-ln(0.8)-ln(0.9) ≈ 1.196; mean loss is about 0.299. Every step predicts a full vocabulary distribution. Computing the loss measures error; it is not differentiation, and the loss stays positive when the correct token has probability below 1. / 5. Backpropagate: apply the chain rule backward through the computation graph. Gradients flow through decoder time steps, the encoder–decoder connection, and encoder time steps. For an RNN this is backpropagation through time (BPTT). Backpropagation computes gradients; it neither translates the sentence in reverse nor directly determines the optimal parameter values. / 6. Optimize: gradient descent and related optimizers update parameters using new = old − learning_rate × gradient. The gradient indicates local sensitivity; the learning rate controls the update size. Continue with the next training batch. / Toy mathematical example: take ŷ = ax²+b, x=2, y=10, initial a=b=1, and learning rate η=0.01. The first forward pass gives ŷ=5. With squared-error loss L=(ŷ−y)², we get L=25 and L(a,b)=(4a+b−10)². Partial derivatives are ∂L/∂a=2(4a+b−10)×4=−40 and ∂L/∂b=2(4a+b−10)=−10. These are gradients, not new parameter values. / The optimizer updates a=1−0.01×(−40)=1.4 and b=1−0.01×(−10)=1.1. The new prediction is ŷ=1.4×2²+1.1=6.7 and the new loss is (6.7−10)²=10.89, down from 25. A large learning rate could instead increase loss. One training sample does not uniquely determine both parameters or guarantee generalization. / Analogy: the toy model input x=2 corresponds to the source sequence; target y=10 corresponds to reference tokens. The predicted scalar corresponds to per-position token probability distributions; a,b correspond to trainable embeddings and encoder, decoder, and output-layer weights θ. The toy model uses squared error; Seq2Seq typically uses cross-entropy. Both rely on gradients and an optimizer. / Training versus inference: training has source and reference target, uses teacher forcing, computes loss and gradients, and updates weights. Inference has only the source; decoding starts with <BOS> and feeds back generated tokens until <EOS> or another stop condition. Ordinary inference does not update weights. / Key principle: the forward pass computes predictions; loss measures error; backpropagation computes gradients; the optimizer updates parameters. Formally: ŷ=f(x;θ), L=Loss(ŷ,y), g=∂L/∂θ, and θ_new=θ_old−ηg. Training repeats until validation results, budget, or a stopping criterion warrant stopping.
        
        Inference: how the model translates a new input
        
        Encoder + Decoder Generation
        
        Encoder is exactly the same as the training.
        
        Input: “I love cat” / │ / ▼ / Tokenizer / │ / ▼ / [1, 2, 3] / │ / ▼ / Embedding / │ / ▼ / x₁ x₂ x₃ / │ / ▼ / RNN Encoder / h₁ = RNN(x₁, h₀) / h₂ = RNN(x₂, h₁) / h₃ = RNN(x₃, h₂) / │ / ▼ / Last hidden state h3
        
        decoder
        
        No loss.
        
        No reverse transmission.
        
        No teachers forced
        
        Inference / │ / ▼ / Input: I love cat / │ / ▼ / Tokenizer / │ / ▼ / Embedding / │ / ▼ / RNN Encoder / │ / ▼ / Last hidden state h3 / │ / ▼ / Initialize Decoder / │ / ▼ / Enter / │ / ▼ / RNN Decoder / │ / ▼ / Linear + Softmax / │ / ▼ / Next Token Probability Distribution / │ / ▼ / Select a Token / │ / ▼ / Is it EOS? / │ │ / No. Yes. / │ │ / ▼ ▼ / End of Token to be generated / Enter the next step / │ ▼ / Output: I love cats. / │ / └──→ RNN Decoder
        
        You can actually compare to Hashmap.
    - Original Transformer
      - Modules
        
        encoder
        
        Self-attension: Multiple location information, which can be calculated in parallel
        
        Traditional
        
        Input: I love cat / ↓ / Tokenizer / ↓ / Embedding / ↓ / x₁ x₂ x₃ / I love cat / ↓ / RNN Encoder / h₁ = RNN(x₁, h₀) / ↓ / h₂ = RNN(x₂, h₁) / ↓ / h₃ = RNN(x₃, h₂) / ↓ / Last hidden state h3 / ↓ / Decoder
        
        Present
        
        Input: I love cat / ↓ / Tokenizer / ↓ / Embedding / ↓ / Organisation / ↓ / x₁ x₂ x₃ / I love cat / \ | / / \ | / / ↓ ↓ ↓ / Self-Attention / ↓ / FPN Front Feed Network / ↓ / z₁ z₂ z₃ / ↓ / Decoder
        
        Attention weights
        
        Why It Is Needed
        
        Because of the different information contributions that Token may make to the current Token, we need a mechanism to determine dynamically which Token should get.
        
        Evolution
        
        No weights
        
        Include weights
        
        QKY
        
        decoder
        
        Masked Self-Attention。
        
        Traditional
        
        Encoder finally hidden state h3 / ↓ / Initialize Decoder / ↓ / Enter RNN→Linear→Softmax / ↓ / “I” is predicted. / ↓ / Enter the real “I” / ↓ / RNN / ↓ / “Love.” / ↓ / Enter real “love” / ↓ / RNN / ↓ / Forecast cat.
        
        Present
        
        Target text: I love the cat / Decoder input: / I love cats. / │ │ │ │ / ▼ ▼ ▼ ▼ / Embedding / ↓ / Organisation / ↓ / Masked Self-Attention / ↓ / Cross-Attention-Encoder Output / ↓ / FFN / ↓ / Linear + Softmax / ↓ / I love cats.
      - Process
        
        Training
        
        Encoder + Decoder + Los + Reverse
        
        Inference
        
        Encoder Encoder Encoder + Decoder Autoregister
    - GT Class Transformer
      - Process
        
        Inference
        
        Prefill
        
        Parallel processing input token
        
        Input: / “Translate I love cat into Chinese:” / │ / ▼ / Tokenizer / │ / ▼ / Embedding / │ / ▼ / Transformer Decoder / Parallel processing input Token / │ / Ideas - Create KV Cache / │ / ▼ / First forecast, Token. / │ / ▼ / I’m…
        
        Decoding
        
        Order output token
        
        Generated: I / │ / ▼ / Transformer Decoder / Using KV Cache / │ / ▼ / Forecast: Love / │ / ▼ / Transformer Decoder / Using KV Cache / │ / ▼ / Forecast: Cat / │ / ▼ / Transformer Decoder / │ / ▼ / Forecast: EOS / │ / ▼ / I love cats.
        
        Training
        
        Use cause and effect mask to predict multiple Tokens, calculate Los and update parameters
      - Modules
        
        Just decoder.
    - Speech
      - https://www.bilibili.com/video/BV1fGeAz6Eie/?vd_source=79c9f80f56384444d88bfb3e4cf579df
Training
- Process
  - Pretraining
    - Ideas - Self-supervised learning
    - Forward
    - Los
    - Backward transmission
    - Optimizer
    - │–output: pre-training basic model
    - Purpose
      - Pre-training allows models to learn language, knowledge and a certain degree of reasoning
  - Post-training
    - Purpose
      - Let models learn how to use these capabilities according to human requirements.
    - Process
      - Oversight fine-tuning
        
        Reliance manual labelling
        
        Tell the model to answer.
        
        Assume that users ask: / What’s the GMP schedule model for Go? / A person provides a high-quality answer: / GMP is a running-time scheduling model for Go, consisting of G (Goroutine), M (Machine) and P (Processor)… / The model adjusts the parameters by learning a lot of such question-and-answer data to make itself better at following instructions. / Nature: Show the model the right answer to imitate learning.
      - Preference Optimization
        
        It’s better to teach model answers.
        
        The same question, the model produces two answers: / Answer A: / The GMP is a schedule model for Go. / Answer B: / The GMP consists of three core components, G, M and P, each with responsibilities for tasks, execution of threads and movement of resources … / The labeler thinks B is better. / The model adjusts the parameters to the preferred data to increase the probability of producing quality answers. / Nature: To make models learn more in line with human preferences by responding more differently.
        
        Techniques
        
        RLHF: Enhanced learning based on human feedback.
        
        There is often a need to train a reward model before optimizing LLM through enhanced learning
        
        LLM Generation Answers / ↓ / Rating of reward models / ↓ / PPO enhanced learning algorithm / ↓ / Update LLM arguments / ↓ / Make the model more inclined to generate high reward answers
        
        DPO: Direct preference for optimization.
        
        LLM was optimized by direct use of human preference data, and independent reward models and enhanced learning processes were omitted.
        
        Problem / Idea-A: Poor. / – Answer B: Better. / ↓ / DPO Loss / ↓ / Update LLM arguments
- Size
  - Three main elements
    - Process
      - (untitled topic)
      - Model, what model do you learn from? Parameters, structure, parameter efficiency
      - Data, what kind of data do you train? Quantity, quality, diversity
      - How much computing resources do you train? GPU Number, Performance, Visibility, Training Duration, FLOPS
    - Scaling Law (the law of scale)(N)Training Token Number(D)Training calculations(C)。
      - Number of tokens in training
        
        The data are three golden laws.
        
        Number
        
        13B means 13 billion parameters.
        
        Quality
        
        Dataset A: contains a large number of advertisements, garbage pages, duplicate text and erroneous knowledge.
        
        Data set B: contains selected technical documents, teaching materials, papers and high-quality web pages.
        
        Diversity
        
        Indicates the amount of knowledge the model has learned
        
        The number of Tokens trained (Training Tokens) refers to the total number of Tokens actually processed by the model during the training, rather than the number of Tokens generated by the model after the training has been completed. / More precisely, it is the number of Tokens actually sent to model training after the original text has been pre-processed by Tokenizer, data cleansing, etc.
      - Model Parameters
        
        13B means 13 billion parameters.
        
        Express the learning ability of the model
      - FLOPs
        
        Calculated resources
        
        Units
        
        FLOP
        
        Total float-point operations required to complete a given task
        
        Reflects training costs
  - Law of Size
    - What It Is
      - Model performance changes according to what pattern when model parameters, training data and calculations are increased.
      - The law for calculating the optimal size and the size of the data set
    - Formula
      - Lulu.
        
        The index, alpha, determines the rate of change.
        
        Input expands 10 times, output changes 10 ^ alpha times
      - Subtopic 2
    - Outsize Push
      - Hyperparameter Migration
        
        There are 10 super-parameters, and we need a two-dimensional set.
      - What It Is
        
        Use of patterns detected in small-model experiments to predict the performance of larger models.
    - Bottleneck
      - Data bottlenecks
        
        High-quality, legally available data that meet training needs are limited, and the marginal cost of continuing to expand data may increase.
      - Electricity bottlenecks
        
        Larger Model + More Training Data / ↓ / More FLOPS / ↓ / More GPU Calculating Time / ↓ / Higher demand for electricity and heat dispersion / ↓ / Limitations on grid capacity, room heat dispersion, equipment cost, etc.
      - Decline in proceeds
        
        Suppose you increase the calculation budget tenfold. / Model performance may be improved, but usually not 10 times. / For example, as the scale expands, Loss may continue to decline, but the improvement is gradually decreasing.
    - Value
      - Reduced trial error costs for large-scale training programmes.
        
        It’s very expensive to practice once. It’s impossible to try all the parameters and data combinations.
1. Inference
- Generation and Output Control
  - Sampling
    - Problems
      - Inconsistency
        
        Same input, different output
        
        Cache
        
        Random Feeds
        
        Enter a little bit the same. The output is completely different.
        
        Phrasing
        
        Memory system
      - Imagination.
        
        Reasons
        
        Self-deception.
        
        Self-supervising how to hallucinate.
        
        Pointer
        
        Overseeing how things are hallucinating.
    - What It Is
      - The language model calculates the probability distribution for the next Token and selects one from the candidate Token according to a strategy.
        
        Token Logit (example) Probability / Cats. 2.30 60%. / Dogs 1.61 30% / Bird 0.51 10%
    - Why It Is Needed
      - The model only gives the probability of the next token, but it’s not decided, so it needs a strategy.
    - What’s the strategy?
      - Greedy.
        
        Every time you choose the most likely.
      - Random
        
        Choose a random one.
    - Parameters to influence policy
      - Temperature
        
        Reduce the probability of common tokens and increase the probability of unusual ones
        
        Temperature actually works in Logits, then goes through Softmax, instead of changing probabilities at will.
      - topk
        
        Pick the top k, other alignment. The rest will continue to be unified.
      - topp
        
        The value of k is dynamic.
    - Subtopic 7
  - Generate Control
    - Max New Tokens
    - EOS (concluded Token)
    - Stop Series
  - Structure Output
    - What It Is
      - Output data in the format desired by users
    - Methodology
      - prompt
        
        Before deduction, tell the model to output according to the specified format
      - Reprocessing
        
        After deduction, parse, repair or validate the results
      - Compute More at Inference Time
        
        Increase the correctness of the format through additional calculations, retesting, candidate validation etc. during or after the reasoning process
      - Binding sampling
        
        Token generation blocking candidates that do not match syntax or Schema
      - Align
        
        Training phase to make models learn to follow specific output formats
- Inference Workflow
  - Prefill (prefill)
    - The model is a one-time process of known input Token, which calculates the context and prepares the phase of the cache for subsequent generation.
      - First token
    - Prefill (prefill) / I love cat / │ │ │ / ▼ ▼ ▼ / x₁ x₂ x₃ / │ │ │ / └────────┼─────────┘ / ▼ / Transformer. / │ / ▼ / h₁ h₂ h₃ / │ / Ideas - Save KV Cache / │ / ▼ / First new forecast Token
  - Decode
    - Gradually calculates the next Token process using known context and previously generated Token.
      - Follow-up token
    - Decode / Round 1: / Context: I love cat / ↓ / Transformer / ↓ / Logits / ↓ / Sampling / ↓ / Yeah. / Round 2: / Context: I love cat / ↓ / Transformer / ↓ / Logits / ↓ / Sampling / ↓ / Cat. / Round 3: / Context: I love cat I like cat / ↓ / Transformer / ↓ / Logits / ↓ / Sampling / ↓ / End Tag

Challenges of Building an LLM

1. Question of expression: What mathematical functions can express reality? / 2. Optimization: How can the appropriate parameters for this function be found efficiently? / 3. Generalization: How can it be ensured that it applies not only to training data but also to unknown data? / Transformer, among other things, offers a very successful solution to the first problem, while its structural characteristics also facilitate large-scale training in the second. / Thus, the basic training algorithm of the nervous network is not difficult to understand, and it is difficult to get this algorithm to really learn the patterns that can be generalized with complex models, big data and limited computing resources.

How to Optimize It

Model Distillation
- What It Is
  - It’s also known as knowledge distillation.
  - Training of student models using monitoring signals provided by teacher models to enable students models to learn about their knowledge or behaviour
  - Core Components
    - Teacher Model (Teacher Model)
      - Provide target behaviour or monitor signals
      - It’s usually a more powerful model.
    - Stuart Model (student model)
      - Knowledge or behaviour in teacher models
      - It’s usually a smaller model.
- Why It Is Needed
  - No model distillation
    - Higher inference costs for direct deployment of large models
    - Small models may not meet mission requirements.
    - Higher cost of manual construction of quality training data
  - Include model distillation
    - Using the skills of teacher models to guide student model training
    - Bringing student models close to teacher models for targeted tasks
    - Reduced cost of inference and consumption of deployed resources
- How to Implement It
  - Overall Workflow
    - Sets the target capacity, selects the teacher-student model, gets the teacher-supervised signal, trains the student model, evaluates distillation effects.
  - Main methodology
    - Respond to distillation
      - Teacher model to generate target impact Response
      - Verify Generating Data
      - Using data for student models
    - Logit distillation
      - Output probability distribution for accessing teacher models
      - The training of students model corresponds to the output distribution of the teacher model
  - Training methods
    - Train students from scratch
    - Fine-tuning based on pre-training models
- How to Evaluate
  - Target mission performance
  - Performance gap with teacher models
  - Inference Latency
  - Inference Costs
  - Panoramic capacity and long tailings performance
- Limitations
  - Not ensuring that the student model reaches the full capacity of the teacher model
  - Could be a teacher’s error.
  - Teachers monitor the quality of signals and influence distillation
  - Need to comply with license for model output data
- Relationship to Synthetic Data
  - Data synthesis
    - Responsible for generating training data
  - Model distillation
    - Responsible for moving the skills of the teacher model to the student model
  - The two can be combined.
    - Teacher model generation data, validation data, student training model.
  - Not all training using synthetic data is distillation.