Inference Optimization

1. What It Is

Improve inference speed and resource efficiency—and reduce serving costs—through model, hardware, and runtime optimizations while maintaining output quality.
Inference
- What It Is
  - Use a trained model to compute an output from a given input
- How Inference Differs from Training
  - Training
    - Forward Propagation
    - Compute the Loss
    - Backpropagation
    - Update Model Parameters
  - Inference
    - Typically involves a forward pass only
    - Does not normally update model parameters
Optimization
- Optimization Layers
  - Model
    - Reduce model size and optimize architecture and computation
  - Hardware
    - Improve compute utilization, memory bandwidth, and hardware efficiency
  - Service Level (Serving)
    - Optimize request scheduling, resource allocation, and batching

2. Why Inference Optimization Is Needed

Performance Constraints
- Foundation-model inference requires substantial compute and memory resources / • Physical deployment faces cost, delay and throughput problems, and therefore there is a need to optimize reasoning
  - Model Size
    - Model weights require substantial device memory
  - Computational Complexity
    - Inference involves substantial matrix computation
  - Concurrent Requests
    - Concurrent users increase compute and memory pressure
  - Long Generated Outputs
    - More generated tokens increase inference latency
    - Increased cost of inference
Where Inference Bottlenecks Come From
- 2.2 Autoregressive Decoding Limits Parallelism
  - What It Is
    - Self-regressive language models usually generate Token one by one
  - Why is it inefficient?
    - The last Token relies on a previously generated Token
    - Unable to calculate all Tokens in parallel with Prefill
  - Example
    - 100 Tokens generated
    - Every Token needs 100 ms.
    - It takes about 10 seconds to decode.

4. How to Optimize It

4.1 Define Performance Goals
- Core objectives
  - Reduce delay
  - Increased throughput
  - Reduce the cost of inference
  - Meeting model quality requirements
- Trade-Offs Between Goals
  - Delays versus resources
    - Lowering the delay in individual requests may require the allocation of additional computing resources
  - Bumping and Delay
    - Increased Batch Size usually helps to increase throughput.
    - But it may increase the waiting time for requests.
  - Costs and performance
    - Increased hardware resources may increase throughput or reduce delays
    - But this will increase deployment and operational costs.
  - Efficacy of reasoning and quality of models
    - Reduced numerical accuracy may reduce computing and memory costs
    - But it could affect the quality of model outputs.
- Define Quality-of-Service Requirements
  - Clarification of requests for delays
  - Clear throughput requirement
  - Clear cost budget
  - Clear model quality requirements
  - Specify the amount to be distributed and the length of the input output
4.2 Choose Performance Metrics
- Latency
  - TTFT（Time To First Token）
    - What It Is
      - Time between submission of request and generation of first output Token
    - Main links
      - Please line up.
      - Schedule and input processing
      - Prefill
      - Enter length
  - TPOT（Time Per Output Token）
    - What It Is
      - After the first output, the average generation time of the subsequent output, Token
    - Main links
      - Decode
      - Memory bandwidth
      - Batch processing and resource movement
  - TBT / ITL
    - What It Is
      - Time interval between adjacent output Token
    - Notes
      - The statistical calibration of different systems may vary.
  - Total Response Time
    - Approximate Formula
      - TTFT+TPOT x (output Token Number - 1)
    - Notes
      - This formula assumes that TOT means the average Token interval after the first Token
      - Actual measurements may also be affected by queuing, network transmission, etc.
  - Delay distribution
    - p50
    - p90
    - p95
    - p99
    - Why It Is Needed
      - Averages may cover up long-tail requests
      - High-grade delay reflects the experience of slower requests
- Throughput
  - What It Is
    - Workload handled systematically within unit time
  - Requesting volume
    - RPS
      - Request per second
    - RPM
      - Request per minute
  - Token throughput
    - Enter Token Bitter
    - Output Token Twist
    - Total Token throughput
  - Notes
    - The request is different from Token’s.
    - Need for harmonization of statistical profiles when comparing different systems
- Goodput
  - What It Is
    - Effective processing of specified quality of service requirements within unit time
  - Example
    - Systems complete 10 requests per second.
    - Only 3 requests met the request for delay
    - Goodput is 3 requests/s if the number of requests is met
  - Why It Is Needed
    - A simple increase in the volume of throughput may increase the number of requests for overtime or delay in exceeding the standard
- Resource Utilization and Efficiency
  - GPU Utilization
    - What It Is
      - GPU active time ratio
    - Notes
      - Not equal to actual computing efficiency
  - MFU（Model FLOP/s Utilization）
    - What It Is
      - Level of utilization of actual model calculations versus theoretical computational capabilities
    - Purpose
      - Computation of performance efficiency by supporting judgement
  - MBU（Model Bandwidth Utilization）
    - What It Is
      - Actual memory bandwidth and theoretical maximum bandwidth That’s right.
    - Formula
      - MBU = actual use of bandwidth / theoretical maximum bandwidth
    - Purpose
      - Help judge memory bandwidth utilization
  - Analytical attention
    - High utilization does not mean business performance Okay.
    - Need for a combination of delay, throughput and resource bottlenecks
    - The calculation limits focus on computing efficiency
    - Focus on bandwidth utilization when memory bandwidth is limited
- Cost
  - Cost per Request
  - Cost per Token
  - Cost per Million Tokens
  - Hardware rental cost
  - Energy and operating costs
  - Notes
    - Need to clarify cost-statistical cycles and distribution modalities
    - Need to analyse actual service costs in conjunction with effective throughput
- Model Quality
  - Why It Is Needed
    - Quantification and cutting may reduce model quality
    - Distillation may not retain the full capacity of the teacher model.
    - Partial optimization may change model behaviour
  - How to Evaluate
    - Reuse chapter 4 assessment methodology
    - Use Same Test Set
    - Comparison of results before and after optimization
    - Check mission success rate
    - Check for accuracy and compliance.
4.3 Identify Performance Bottlenecks
- Diagnostic Principles
  - See if performance indicators meet the target first
  - Repositioning time consumption and resource costs
  - Key bottlenecks identified through measurements and experiments
  - Do not judge bottlenecks by technical name or appearance
- Compute-Bound
  - What It Is
    - Implementation time is mainly limited by computing capacity
    - The calculation unit is the main bottleneck
  - Typical scene
    - The LLM Prefill stage may normally be subject to computational limitations.
    - Inference in certain image-generation models
  - Analysis Directions
    - Check Calculator
    - Check to calculate implementation efficiency
    - Check indicators such as MFU
  - Potential Improvements
    - Kernel Optimization
    - Compiler Optimization
    - Quantitative
    - Hardware computing capability optimization
  - Notes
    - Prefil is not always limited in all tasks.
- Memory-Bandwidth-Bound
  - What It Is
    - The execution time is mainly limited by the speed of data transmission
    - Calculating units need to wait for model weights or other data from memory Add Load
  - Typical scene
    - LLM Decode is usually subject to memory bandwidth limitations
    - Especially the lower Batch Size self-return code.
  - Analysis Directions
    - Check to re-read expenses
    - Check KV Cache for expenses
    - Check memory bandwidth usage
  - Potential Improvements
    - Quantization
    - Optimizing memory access
    - Improving the utilization of resources through the improvement of Batch Size
    - Kernel and Count perform optimization
  - Notes
    - Increased catch Size and possible additional delay and KV Cache occupation
- Memory Capacity
  - What It Is
    - Model weights, KV Cache and other running data cannot be accommodated in available memory
  - Typical problem.
    - OOM
    - The model cannot be fully loaded
    - Large KV Cache for context and high combined request
  - Analysis Directions
    - Analyse model weighting
    - Analyse KV Cache occupation
    - Analysis of temporary data occupancy at running time
    - Analysis of memory growth resulting from concurring requests
  - Potential Improvements
    - Quantization
    - Distillation
    - Pruning
    - KV Cache Quantization
    - KV Cache Compression
    - PagedAttention
    - Model Parallelism
    - Expand hardware memory capacity
  - Notes
    - PagedAttention mainly improves KV Cache management and distribution efficiency
    - It does not automatically reduce the amount of KV data required per Token.
- Queueing and Scheduling Bottlenecks
  - What It Is
    - Request awaiting dispatch or execution of resources
  - Possible causes
    - Request arrival rate exceeded system processing capability
    - Batch’s waiting too long.
    - Prefill and Decode compete for resources.
    - Control strategy is not reasonable
  - Potential Improvements
    - Dynamic Batching
    - Continuous Batching
    - Prefill/Decode Disaggregation
    - Replica Parallelism
- Compute and Communication Bottlenecks
  - What It Is
    - Inter-equipment communications or synchronized costs affect efficiency of implementation
  - Possible causes
    - Inter-equipment communication for Tensor Parallelism
    - Pipeline Parallelism ’ s Stage Transfer
    - Data transfer and synchronization in distributed systems
  - Potential Improvements
    - Adjust Parallel Policy
    - Optimizing communications and calculating overlaps
    - Evaluate equipment bandwidth interconnection
4.4 Select Optimization Techniques
- Model Weights Consume Too Much Memory
  - First Determine
    - Could not load the model
    - Whether weights are overrepresented
    - Whether weight reading causes memory bandwidth bottlenecks
  - Candidate Technologies
    - Quantization
      - Decrease value accuracy
      - Less weight storage expenses
      - Potential reduction in memory bandwidth requirements
      - Needs assessment model quality
    - Distillation
      - Model for training less capable students
      - The deployment of smaller student models reduces the cost of inference.
      - Need extra training.
      - It may not be possible to retain the full capacity of the teacher model.
    - Pruning
      - Remove Partial Parameters or Structures
      - Possible reduction in model size and calculation costs
      - Need to assess whether thin structures are efficiently supported by hardware
  - Notes
    - Distillation is not a direct compression of existing model weights
    - The fact that unstructured cuttings create a thin weight does not mean that reasoning is bound to accelerate.
- Decoding Latency Is Too High
  - First Determine
    - Whether TPOT exceeds target
    - Main bottlenecks are memory bandwidth or computing
    - Whether queuing or movement costs exist
    - Whether it’s affected by KV Cache’s reading expenses
  - Candidate Technologies
    - Speculative Decoding
      - Generate a candidate Token using a draft model
      - Verify candidate sequences by target model
      - Reduced number of sequential decodes of target models
      - Proceeds depend on the cost of the draft and the acceptance of the candidate Token
    - Parallel Decoding
      - Try to predict multiple future Tokens in parallel
      - Determine acceptable output by authentication or filtering
      - Representative technology includes Lookahead Decoing and Medusa
      - Different programmes may require additional training or model modifications
    - Attention Optimization
      - Consider when calculating attention or KV Cache visits constitute bottlenecks
      - Specific technologies need to be selected based on actual bottlenecks
  - Notes
    - Hypothetical decoders and parallel decoders do not apply to all loads.
    - TPOT, throughput, cost and output quality should be assessed simultaneously
- Long Contexts Cause Excessive KV-Cache Memory Use
  - First Determine
    - Whether KV Cache is the main RAM source
    - The question is whether memory capacity is inadequate or is memory access too expensive.
    - Whether or not to keep the whole historical context
  - Candidate Technologies
    - KV Cache Quantization
      - Reduce Cache Numeracy
      - Reduce cache occupancy
      - Need to assess accuracy loss
    - KV Cache Compression
      - Compress Cache Contents
      - Reduce Cache Storage Needs
    - Selective KV Cache
      - Selective retention of cache information
      - Need to assess the impact of discarded information on model quality
    - PagedAttention
      - Manage KV Cache using non-continuous memory blocks
      - Reduction of memory debris
      - Improving the efficiency of cache distribution and management
      - Support for more flexible cache sharing
    - Attention structure optimized
      - Local Windowed Attention
        
        Limit attention window
        
        Reduce historical information that needs to be processed or cached
      - Multi-Query Attention（MQA）
        
        Multiple Query heads share a set of Key and Valle
      - Grouped-Query Attention（GQA）
        
        Multiple Query Head Group Sharing Key and Valle
      - Cross-Layer Attention
        
        Part of the attention information shared across specific structures
  - Notes
    - Architecture optimization usually needs to be addressed during model design, training or fine-tuning
    - PagedAttention mainly optimizes memory management, not equivalent to KV Cache compression
- Repeated Prompts Waste Computation
  - Use Cases
    - Multiple requests shareSystem Prompt
    - Multiple requests repeat long prefix
    - Multi-cycle dialogues include reusable historical prefixes
  - Candidate Technologies
    - Prompt Caching
      - Prefix Caching or Context Caching
      - Recognize reusable Prompt prefix
      - Calculator for Cache
      - Reuse new application after cache has been completed
      - Calculate only those parts that are not cached
  - Proceeds
    - Reduce repetition of Prefill calculations
    - Lower First Token Delay
    - Reduce the calculated cost of duplicate input
  - Limitations
    - Need Cachespace
    - Need to manage the Cache Life Cycle
    - Gains depend on the CAL and reusable range
- Throughput Is Too Low
  - First Determine
    - Requesting or Token’s not enough.
    - Whether the system is subject to computing, memory or queue restrictions
    - Does the current Batch Size make sense?
    - Distribution of request length and arrival rate
  - Candidate Technologies
    - Batching
      - Static Batching
        
        A fixed number of requests make up the Batch.
        
        Make it simple.
        
        We may need to wait for the approval.
      - Dynamic Batching
        
        Set maxbatch Size
        
        Set maximum waiting time
        
        Implementation when conditions are met
        
        Between batching efficiency and waiting time Right Balance
    - Continuous Batching
      - Development of requests for accession and removal during the implementation of reasoning
      - Release the location after completion of the request
      - Allow new requests to be added to subsequent implementation steps
      - Reduced congestion between requests
      - Increased efficiency in the use of hardware resources
    - Replica Parallelism
      - Deployment of complete model copies
      - Sharing of different requests
      - Increased co-processing capacity
      - Additional hardware resources required
  - Notes
    - Batch processing does not necessarily improve all types of delay
    - Goodput and high-grade delay should be checked simultaneously.
- Prefill and Decode compete for resources.
  - Problems
    - Prefill and Decode have different resource requirements
    - It could interfere with each other.
  - Candidate Technologies
    - Prefill/Decode Disaggregation
      - Prefil runs in independent cases
      - Decode runs in a stand-alone instance
      - Transfer necessary KV Cache intermediate state
  - Proceeds
    - Reduced resource disruption between phases
    - Optimizing TTFT and TOT
    - Facilitating the independent allocation of resources
  - Cost
    - Increased communications costs
    - Increase schedule complexity
    - Need to consider KV Cache transfer costs
  - Applicability
    - Need to combine load features with system architecture assessment
- Unable to deploy model for single device
  - First Determine
    - Whether the model weight exceeds the single device memory
    - Whether KV Cache and running time data exceed memory
    - The goal is to support model deployment or increase throughput.
  - Candidate Technologies
    - Tensor Parallelism
      - Split screen operations to multiple devices
      - Support for larger model deployment
      - Possible reduction of partial computation delay
      - Cost of inter-equipment communications
    - Pipeline Parallelism
      - Models divided into multiple phases
      - Different stages of deployment of equipment
      - Overlap can be calculated through the Micro-batch implementation phase
      - Support for large model deployment
      - Possible additional request delay
      - Irrelevance of communication and flow lines during life
    - Quantization
      - Lower model weighting
      - Helps model load single devices when quality requirements are met
  - Notes
    - In parallel, the model first addresses the issue of model separation and deployment
    - Whether or not to increase the delayed or swallowed amount depends on parallel strategy and communication costs
- High complication leads to insufficient resources
  - First Determine
    - Is there a queue?
    - GPU computational capacity constraints
    - Are there memory capacity limitations?
    - Is there a control efficiency problem?
  - Candidate Technologies
    - Replica Parallelism
      - Add Model Copy
      - Share request load
      - Improved overall processing capacity
      - Increased hardware costs
    - Continuous Batching
      - Increased efficiency in the movement and batch processing of requests
      - Reduced congestion between requests
    - PagedAttention
      - Improved memory management efficiency of KV Cache
      - Possible higher number of concurrent requests supported
  - Notes
    - High side hair is a task load feature, not a single hardware bottle The neck.
    - Need for a combination of calculation, memory, queuing and comprehensive cost judgement
- Calculating nuclear inefficiency
  - First Determine
    - Calculating underutilization of resources
    - Is there a significant start-up cost for the algorithm execution?
    - Existence of redundant intermediate data reading and writing
    - Existence of inefficient memory access mode
  - Candidate Technologies
    - Kernel Optimization
      - Vectorization
        
        Process multiple data elements simultaneously
      - Parallelization
        
        Distribution of independent calculations to multiple implementation units
      - Loop Tiling
        
        Calculate fractions based on cache and memory layout adjustments
      - Operator Fusion
        
        Merge Multiple Calculator Operations
        
        Reduce intermediate data reading and writing and algorithm start-up costs
    - Compiler Optimization
      - Lowering
      - Calculator Conversion
      - Select or generate optimized kernels
      - Align implementation with target hardware
    - Relevant technologies
      - CUDA
      - Triton
      - ROCm
      - torch.compile
      - XLA
      - TensorRT
      - Apache TVM
      - MLIR
  - Notes
    - Kernel’s the bottom computing unit.
    - Compiler is responsible for targeting algorithms or algorithms for execution.
    - Specific optimization depends on model, hardware and compilation results
- Other workload issues
  - Large in context
    - Analyzing attention to calculate costs with KV Cache
    - Consider focus computing and architecture optimization
  - Insufficient capacity to process requests
    - Analysis of the movement of requests, resource utilization and distribution of requests
    - Consider batching, parallel deployment and build-up
  - Overcost
    - Analysis of unit requests and unit Token costs
    - Consider quantification, batching, cache and resource allocation
4.5 Validation of optimized effects
- Establishing a baseline before optimization
  - Record TTFT
  - Recording TPAT and total response time
  - Recording of delayed fractions p50, p95, p99, etc.
  - Recording requests for throughput and token throughput
  - Record Goodput
  - Record resource indicators such as GPU Utilisation, MFU and MBU
  - Record unit request or unit Token cost
  - Document the results of the quality assessment of the model
- Control of experimental conditions
  - Use same or comparable test data
  - Control input length and output length
  - Control the combined volume and request arrival rate
  - Clear hardware configuration and model version
  - Clarify statistical windows and indicator calibres
- Comparison of results before and after optimization
  - Reduced delay
  - Increased throughput
  - Would Goodput Increase
  - Improved use of resources
  - Cost reductions
  - Model quality satisfied
- Check side effects
  - Whether to add a high-grade delay
  - Increase memory occupancy
  - Additional communication or movement costs
  - Decreasing the quality of the model
  - Whether to introduce new performance bottlenecks
- Judge whether or not the optimization is successful
  - Improvements in pre-existing bottlenecks
  - Target indicators met
  - Introduction of unacceptable cost or quality loss
  - Need for continued analysis of new bottlenecks