Fine-Tuning

What It Is

• Continue training some or all parameters of a pretrained foundation model so it performs better on target tasks.
• A form of transfer learning.
• How Fine-Tuning Differs from Prompting and RAG
  • Prompt Engineering
    • Improve responses by designing better inputs, without changing model weights
  • RAG
    • Retrieve relevant external information without changing model weights
  • Fine-Tuning
    • Update model weights using additional training

Why It Is Needed

• Limitations of General-Purpose Models
  • Insufficient capability on domain-specific tasks
  • Poor performance on particular instructions or tasks
  • The output style or format does not meet requirements
  • The model may exhibit unwanted biases
• Benefits of Fine-Tuning
  • Improve performance on specific tasks
  • Improve model behavior and instruction following
  • Enable smaller models to meet domain-specific requirements
  • Reduce ongoing inference costs in suitable scenarios
• When is fine-tuning appropriate?
  • Prioritize Prompt Engineering
  • If the problem is missing or outdated external knowledge
    • Prioritize RAG
  • If behavioural or capacity problems persist
    • Consider fine-tuning
  • Evaluate the Trade-Offs
    • Performance enhancement
    • Cost of Training Data
    • Compute and Training Costs
    • Deployment and Maintenance Costs
• Risk of fine-tuning
  • Other mission capabilities may be degraded
  • It could be catastrophic.
  • Need for quality training data
  • The base model update may reduce the value of fine-tuning.

How to Implement Fine-Tuning

• Overall Workflow
  • 1. Targeting fine-tuning
  • 2. Selection of the base model
  • 3. Preparation of training data
  • Selection of fine-tuning methods
  • 5. Configure the training environment and super-parameters
  • Implementation fine-tuning
  • 7. Assessing the effect of fine-tuning
  • 8. Deployment or continued optimization
• Core Components
  • Base Model
    • What It Is
      • Models that have been pre- or post-trained
  • Training Dataset
    • What It Is
      • Data to adjust model parameters
    • Chapter 8
  • LosFunction (loss function)
    • What It Is
      • Differences between model projections and training objectives
  • Optimizer (optimizer)
    • What It Is
      • Update training parameters based on gradient
    • Common algorithms
      • SGD
      • Adam
  • Training Runtume
    • Responsible for model loading, computing, parameter updating and resource management
• Moderate
  • Classification by training target
    • Continued Pre-training
      • Continued self-monitoring training on the use of field data
    • SFT (oversight fine-tuning)
      • Training using input and target output
    • Preference Fine-twinning
      • Adjust model behaviour using preferred data
    • Special objectives
      • Infilling Fine-tuning
      • Long-context Fine-twining
  • Category by parameter update
    • Full Fine-tuning
      • Update all model parameters
    • Partial Fine-twining (partial fine-tuning)
      • Only update part of the original parameter
    • PEFT
      • Adaptation model through a small number of training parameters
      • Adapter
      • Soft Prompt
      • LoRA
        • Updates with learning weights using low-twilight matrix
      • QLoRA
        • Combining quantitative base model with LoRA
• Detailed Execution Flow
  • Load Base Model
  • Identification of trainingable parameters
  • Load training data
  • Forward Pass (forward transmission)
    • Calculate Model Output
  • Calculating Los
    • Comparison of projections and training objectives
  • Backward Pass (reverse transmission)
    • Calculate gradient
  • Optimizer update parameter
  • Repeat training until the conditions for cessation are met
  • Save model or Adapter

How to Optimize Fine-Tuning

• Why do you need optimization?
  • Full fine-tuning takes a lot of memory.
  • Higher training time and costing
• Memory bottlenecks
  • Main memory sources
    • Model weight
    • Gradient
    • Optimizer status
    • Activate Value
• Memory Optimization
  • Reduction of trainingable parameters
    • PEFT
    • LoRA
  • Reduce Numeric Precision
    • Quantification
    • Mixed Precision (mixed precision)
  • Group Optimization
    • QLoRA
  • Memory Unmounting
    • CPU Offloading
• Trained overparameter optimization
  • Learning Rate (learning rate)
  • Batch Size ( Batch size)
  • Gradient Accumulation
  • Epoch (Train Wheels)
  • Prompt Los Weight
• Multi-task model customization
  • Also fine-tuning
    • Training using data from multiple missions
  • Order fine-tune
    • Training on different tasks.
    • We need to be careful of the catastrophic oblivion.
  • Model Merging
    • Parameters for combining different models or Adapter
    • It’s not like fine-tuning.
    • Complementary model customization techniques

How to Evaluate Fine-Tuning

• Reuse chapter 4 assessment methodology
• Quality assessment
  • Improved performance of target missions
  • Deterioration of capacity for other missions
  • Compliance with operational quality requirements
• Training process assessment
  • Training Loss
  • Validation Loss
  • Has there been a match?
• Cost assessment
  • Training costs
  • Inference Costs
  • Maintenance costs
• Comparative assessment
  • Original base model
  • Modified Model
  • Prompt Engineering
  • RAG Programme