ThinkerQAQ

NOTE

Dataset Engineering

Survey the dataset engineering lifecycle: requirements, acquisition, synthesis, cleaning, validation, and quality control.

AIUpdated 12 min read

Dataset Engineering

1. What It Is

  • Acquire, generate, process, and validate data against training objectives to build high-quality datasets that improve model performance.
    • Data requirements and objectives for different stages of training
      • Pretraining
        • Data Format
          • Raw text and related source data
        • Purpose
          • Help the model acquire general knowledge and foundational abilities
      • Post-training
        • Supervised Fine-Tuning (SFT)
          • Data Format
            • Instruction–Response Pairs
          • Purpose
            • Teach the model to follow instructions and perform specified tasks
        • Preference Optimization
          • Data Format
            • Prompt, preferred response, and rejected response
          • Purpose
            • Align model responses more closely with human preferences
        • Reward Model Training
          • Data Format
            • Prompt, preferred response, and rejected response
            • Alternatively: prompt, response, and quality rating
          • Purpose
            • Train a reward model to score response quality
  • The central idea is data-centric AI
    • Data-Centric AI
      • Improve model performance by optimizing training data
    • Model-Centric AI
      • Improve model performance by optimizing model architecture, training methods and hyperparameters

4. How to Optimize It

    1. Data Curation
    • Optimizing data quality
      • Refine labelling guidelines
      • Manual review
      • AI support label
      • Automation validation
    • Optimizing data coverage
      • Recognize the missing scene
      • Targeted complementary samples
      • Targeted synthetic data
    • Optimizing data volume
      • Small sample experiment.
      • Whether to add data according to the learning curve
      • Use data clipping to screen high-value samples
    1. Data Acquisition and Synthesis
    • Reduce Data Acquisition Costs
      • Prioritize reuse of existing data
      • Combining manual and AI generation data
      • Continuous accumulation of data through real business feedback
    • Limitations of synthetic data
      • Quality Issues
        • Could contain error of fact or error label
      • Surface imitation
        • Models may learn only expression, but not real ability.
      • Model Collapse
        • What It Is
          • Recursive use of models to generate data training follow-up models, which may lead to data distribution degradation and model performance decline
        • Reasons
          • Models tend to generate high probability content
          • Rare patterns may disappear over time.
        • Mitigation
          • Mixed use of real and synthetic data
          • Validate synthetic data quality
          • Retain diversity in data distribution
        • Notes
          • If you don’t use synthetic data, you’ll have to cause the model to collapse.
      • Data traceability
        • Source of raw data may not be transparent
        • Possible copyright risk
        • Possible introduction of baseline test contamination
  • Data Processing
    • Improve Data Processing Efficiency
      • First perform low-cost and good sift-out operations
      • Small-scale pilot before full-scale processing
      • Keep raw data
      • Avoid directly overriding raw data

2. Why It Is Needed

  • Background
    • Models for learning knowledge and behaviour through training data
    • Training data quality and distribution impact model final performance
  • Problems Without Dataset Engineering
    • Inconsistent Data Quality
      • There may be an error in the original data
      • Repeated data may introduce deviations
      • Discrepancies in labelling may affect model learning
    • Coverage Gaps
      • Real data may concentrate on a few common scenes.
      • Lack of samples of border scenes and rare missions
      • Models may not be able to fully learn about target mission behaviour
    • Difficulty Acquiring Data
      • Scarcity of data in professional areas
      • High cost of manual labelling
      • Real data may be restricted by privacy and copyright
    • Raw data often needs cleaning and validation before training
      • Need to meet specific training objectives
      • Need for data formats that meet model requirements
  • What Dataset Engineering Improves
    • Improving data quality
      • Access to reliable data through inspections, weighting, cleaning and labelling
    • Improve data coverage
      • Complementary target scenario through data mapping and data synthesis
    • Meeting training requirements
      • Build the corresponding data formats according to training objectives
    • Control of data costs
      • Rights between data quality, coverage, quantity and budget Balance

3. How to Implement It

  • Overall Workflow
    • Set Data Requirements → Acquire or Synthesize Data → Process Data → Validate the Result
    • Train Models and Evaluate the Results
  • Detailed Workflow
      1. Data Curation
      • Responsibilities
        • Quality, diversity and quantity requirements for data sets based on model training objectives
      • Quality
        • Metrics
          • Relevance and mission compliance rate
            • % of samples meeting the target mandate and expected behaviour
          • Correct Rate
            • Proportion of samples with the correct content or label
          • Coherence
            • Level of consistency in the representation of the same samples among the various labelers
          • Format pass rate
            • Proportion of samples meeting data format requirements
          • Repeat Rate
            • Percentage of duplicated or nearly duplicated samples
          • Compliance rate
            • Proportion of samples meeting privacy, copyright, etc.
        • Measurement Methods
          • Rule Verify
          • Manual sampling
          • AI support assessment
          • To retest.
      • Diversity / Diversity
        • Metrics
          • Area coverage
            • Number of target scenes/ total target scenes covered
          • Sample distribution
            • Number and percentage of samples from each scene
          • Area coverage adequacy
            • Whether critical scenes have enough effective samples
        • Measurement Methods
          • Definition of scene classification by training target
          • Number of samples by scene
          • Identification of missing or inadequately sampled scenes
      • Quantity
        • Metrics
          • Number of active samples
            • Number of samples cleared, removed and reweighted
          • Learning Curve
            • Model performance changes at different data scales
          • Marginal gains
            • Performance improvements resulting from increased data
        • Measurement Methods
          • Number of valid statistical samples
          • Training experiments using data sets of different sizes
          • Draw a learning curve
          • Integrated scale increases with data costs
      1. Data Acquisition and Synthesis
      • What It Is
        • Obtaining or generating data to meet training objectives
      • How to access data
        • Available data
          • Public data
            • Hugging Face Datasets
            • Kaggle
            • Google Dataset Search
          • Proprietary Data
            • Enterprise Internal Documents
            • Data on professional areas
          • Real business data
            • User Input
            • System run data
            • User feedback
        • Data enhancement and data synthesis
          • What It Is
            • Data Enhancement
              • Create a new variant based on existing real data
            • Data Synthesis
              • Artificial generation of data to supplement or replace real data
          • Why It Is Needed
            • Increase in data volume
            • Increase data coverage
            • Improving data quality in selected scenarios
            • Reduce access to real data
          • How to Implement It
            • Data Enhancement
                1. Rules-based
                • Principles
                  • Change original samples according to predefined rules
                • Technical approach
                  • Rotate and Crop Images
                  • Dictionary driven synonym replacement
                  • Template replacement
                  • Perturbation
                • Characteristics
                  • Simple, low-cost, highly manageable
                  • Flexibility and diversity are subject to rules
                1. Based on AI
                • Principles
                  • Use AI model to understand and convert original samples
                • Technical approach
                  • Paraphrasing
                  • Translation
                  • Back-translation
                • Characteristics
                  • High flexibility and diversity of expression
                  • May cause semantic drift or error
            • Data Synthesis
              • Generation Method
                  1. Rules-based
                  • Principles
                    • Construct data directly by predefined rules or templates
                  • Techniques
                    • Template Generation
                    • Random Field Fill
                    • Faker
                  1. Simulation
                  • Principles
                    • Data generated by modelling interactions and state changes in the environment
                  • Techniques
                    • Environmental modelling
                    • Conduct
                    • Status Update
                    • Results validation
                  1. Based on AI
                  • Principles
                    • Production of training data using AI model
                  • Techniques
                    • Self-Instruct
                    • Instruction Backtranslation
                    • Self-play
              • Type of data generated
                • Command Data
                  • For SFT
                • Prefer Data
                  • Training for preference optimization or reward models
                • Data on tool use
                  • Tool mobilization capacity for training models
              • Notes
                • Three methods of generation can be used in combination
                • The same type of data can be generated in different ways
      • Data labeling
        • What It Is
          • Provide mission-compliant labels or target outputs for samples
        • Implementation Approaches
          • Manual Notation
          • AI support label
        • Annotation Guides
          • Define correct response and high quality sound Response
          • Provide examples of right and wrong
          • Ensure consistency in labelling
          • Evaluation criteria for replicating chapter 4
    • Data Processing
      • Responsibilities
        • Processing raw data into data that meets model training requirements
      • Overall Workflow
        • Data check, poach, poach, poach, poach, poach, poach.
        • The order of implementation can be adjusted to data characteristics and processing costs
      • Data Implementation
        • What It Is
          • Get to the data first.
        • Why It Is Needed
          • The quality and distribution of raw data does not make it possible to judge what follow-up needs to be done
          • Need for early detection of data anomalies, identification of conflicts and potential risks
        • Implementation Methods
          • Data source checks
            • Principles
              • Understanding data sources, processing history and use limits through metadata
            • Practical Methods
              • Check data sources
                • Open data, internal data, user data or synthetic data
              • Check data processing history
                • Has it been heavy, cleaned or labelled
              • Check use limits
                • Copyright, privacy and data licensing
            1. Statistical analysis (Statistical Analysis)
            • Principles
              • Understanding the size, structure and distribution of data sets through descriptive statistics
            • Practical Methods
              • Quantity statistics
                • Number of statistical samples and number of Tokens
              • Length Distribution
                • Statistical input, response length
                • Short or long samples were identified by histograms, fractions
              • Category distribution
                • Type of statistical task, subject area, language, etc.
                • Identification of categories with insufficient or over-concentration coverage
              • Group Statistics
                • Analysis by source, time, dimension, etc.
                • Identification of differences in data quality between groups
            1. Unusual detection
            • Principles
              • Identification of samples that clearly deviate from normal data distribution or expected rules
            • Practical Methods
              • Rule Test
                • Check missing fields, abnormal length and illegal extraction
              • Statistical anomaly detection
                • Identification of abnormal scores, abnormal Token length, etc.
              • Mark Conflict Check
                • Identify conflicting labels for identical or highly similar input
                • Analyse whether there is an error in labelling or reasonable context difference
            1. Manual Inspection
            • Principles
              • Identification of problems in automated statistics through manual reading of samples
            • Practical Methods
              • Random sample
                • Random selection of samples from data collection
              • Layer sample
                • Samples from different missions, sources and quality levels
              • Manual review
                • Check if content, labels and target behavior is reasonable
                • Verification of certain factual elements
        1. Data Development
        • What It Is
          • Elimination of unnecessary duplication
        • Why It Is Needed
          • Duplication of data to distort training distribution and introduce deviations
          • Reducing waste of training resources
          • Prevention of data contamination in training and assessment sets
        • Repeat type
          • Full repetition.
            • It’s exactly the same.
          • Approximate repetition
            • The content is highly overlapping, with only minor differences
          • Semantic Repeat
            • Different text, but a highly similar meaning
        • Implementation Methods
            1. Pairwise Comparison
            • Principles
              • Calculates the similarity between the two samples, judging by the threshold whether or not to repeat
            • Practical Methods
              • Exact Matching
                • Directly compared standardized sample content
                • Applies to full repetition
              • N-gram Similarity
                • Split text into consecutive n-words or segments of characters
                • Compare the degree of overlap between the segments in the two samples
                • Applies to Appearance Repeat
              • Fuzzy Matching
                • Calculation of text differences by means of indicators such as editing distance
                • Applicable to small number of characters or term changes
              • Semantic Similarity
                • Compare the similarity of the meaning expressed in the text
                • Text applicable to expressions of different but similar meanings
            • Limitations
              • The cost of comparison for all samples is normally O(N2)
              • Large data sets need to reduce the number of candidates for comparison
            1. Hashing (Hashi)
            • Principles
              • Map the sample to a Hash value or signature, and quickly locate data that may be duplicated
            • Practical Methods
              • Normal Hash
                • Standardize samples as necessary
                • Calculating Sample Hash
                • Identification of duplicate candidates through the Hashi list
                • Check the actual content of the Hashi match results
                • Applies to full repetition
              • MinHash
                • Principles
                  • Use a short signature to estimate the similarity of the two collections
                • Execution Flow
                  • Convert text to n-gram
                  • Calculate MinHash Signature
                  • Filter high-symmetry candidates by signing
                  • Based on the need for further comparison
                • Applicable To
                  • Large text close to heavy
              • Bloom Filter (Bron filter)
                • Principles
                  • Use digits and several Hash functions to determine whether elements are possible
                • Execution Flow
                  • Samples are in place through multiple Hash functions Group
                  • Set the corresponding position when inserting
                  • Check these positions when searching
                • Characteristics
                  • Lower space cost
                  • Allow fake positive
                  • Possible outcomes cannot be directly considered as identifying duplication
                • Applicable To
                  • Quick Repeat Candidate Test for Streaming Data
            1. Vector expression and reduction
            • Principles
              • Map text to vector space to find samples close to semantics by distance or similarity
            • Practical Methods
              • Embedding
                • Convert text to vector using vector models
              • Vector Similarity
                • Compare vectors using indicators such as cosine similarity
              • Vector Search
                • Use an approximate neighbour search to find a similar candidate
              • Demensionality Reduction
                • Lower vector dimension as required
                • Reduced cost of subsequent calculations and storage
              • Intelligent assistants
                • Delete based on similarity threshold and actual semantics
            • Applicable To
              • Semantic Repeat
            • Limitations
              • Semantic similarity does not necessarily mean redundancy.
              • You can’t just remove all samples on the basis of similarity.
        1. Data Clearing & Filtering
        • What It Is
          • Keep Valuable Data
        • Why It Is Needed
          • Original data may contain noise, error or irrelevant content
          • There may be privacy, copyright or security problems with raw data
          • Low-value samples may increase training costs without improving model performance
        • Implementation Methods
          • Data Cleaning
            • Principles
              • Fix or remove non-compliant elements in the sample
            • Practical Methods
              • Constraints and Boundaries
                • Remove non-HTML tab
                • Remove redundant Markdown tag
                • Handle abnormal blank characters and obscurities
              • Data standardization
                • Harmonization of field names, coding and content formats by task
              • Non-compliance with data processing
                • Identification of personal sensitive information (PII)
                • Can not delete folder: %s: No such folder
                • Remove data that does not meet licensing and security requirements
            • Notes
              • Can’t clean up what models actually need to learn.
              • For example, the code model may need to retain the Markdown code block and HTML syntax.
            1. Data filtering
            • Principles
              • Whether the sample should be retained according to quality criteria
            • Practical Methods
              • Rule-based filtering
                • Filter samples using predefined conditions
                • For example, missing required fields, abnormal length, misformatting
              • Functional correctness verification
                • Whether or not the sample output is correct through an enforceable test
                • For example, perform code and run cell tests
              • AI-assisted filter
                • Use models to judge relevance, mission conformity and content quality
                • Screening of samples based on ratings and thresholds
              • Manual audit
                • Samples where inspection rules or AI cannot be reliably judged
                • Review of border and disputed samples
            1. Data Prening
            • Why It Is Needed
              • The size of the data may exceed the training budget
              • Some samples are of low training value
            • What It Is
              • Select a more valuable set of training samples in existing data collections
            • Implementation Methods
              • Active Learning
                • Principles
                  • Priority is given to samples that are expected to best contribute to model learning
                • Typical method
                  • Sample selection based on model uncertainty
                  • A balance between representation and diversity
                • Use Cases
                  • Select a sample to be marked under a limited budget
              • Importance Sampling
                • Principles
                  • Distribution of the probability of sampling based on the importance of the sample
                • Execution Flow
                  • Estimated sample importance
                  • Set sampling weights
                  • Priority sampling of high-value samples
                • Notes
                  • Material sampling does not necessarily require permanent removal of low weight samples
              • Screening based on sample quality or value
                • Calculate mass scores for samples
                • Select a subset according to budget and coverage requirements
            • Notes
              • Cutting could result in a long tail sample missing.
              • Need to balance quality, diversity and quantity
        1. Data Formatting
        • What It Is
          • Convert to training format
        • Why It Is Needed
          • Training interface for raw data structures that may not be in line with target models
          • Different training missions require different forms of data organization
          • Mismatched training and inference formats may affect model performance
        • Implementation Methods
            1. Establishment of training data structure
            • Principles
              • Organize input and target output according to training target
            • Practical Methods
              • Pretraining
                • Use original sequences like text
              • SFT
                • Organizational commands and targets Response
              • Prefer training.
                • Organisational hints, preference responses, etc.
            1. Fit to Chat Template (dialogue template)
            • Principles
              • Organize messages according to the dialogue format specified in the target model
            • Practical Methods
              • Instructions
                • System
                • User
                • Assistant
                • Organisation Tool Call Message by Model
              • Chat Template with Target Model
              • Sequence structured messages into training input
            • Example
              • User: Read config.yaml
              • Assistant: Call read file and upload path parameters
            1. Fit Tokenizer
            • Principles
              • Convert Tokenizer input to Token ID using target compatible model
            • Practical Methods
              • Loading Tokenizer for Target Model
              • Check Token Length
              • Interception or segmenting as required by training
              • Tokenization pre-completed according to the training framework
            1. Handling special Token
            • Principles
              • Mark message boundaries or other structural information by special Token information specified in the model
            • Practical Methods
              • Intelligent assistants
              • Process Serial End Mark (EOS)
              • Call for associated tags according to the model processing tool
            • Notes
              • Use Chat Template to avoid adding special Tokens manually
            1. Format validity verification
            • Check mandatory fields
            • Check character, message sequence and special Token
            • Check if the data can be properly analyzed in the training framework.
            • Check for compatibility of training and inference input formats
      • Engineering practice attention
        • Keep raw data before processing
        • Use a small sample to validate the script first
        • Reordered processing according to cost and data characteristics
        • Record changes in the number and quality of samples before and after processing
      1. Data Validation
      • Responsibilities
        • Validate data for pre-defined quality
      • Overall Workflow
        • Determines the acceptance standards → perform validation → statistical results → judgement eligibility → feedback repair
      • Data Quality Validation
        • What It Is
          • Check whether data content, labels and formats meet predefined quality standards
        • Why It Is Needed
          • Data processing doesn’t mean data is correct.
          • There may be errors in manual tagging and AI synthesis data
          • Error data can lead to model learning error behavior
        • Implementation Methods
          • Rule-based Validation
            • Principles
              • Automatic determination of whether the sample is in accordance with predefined rules
            • Practical Methods
              • Check if the required fields exist
              • Check JSON Schema or data type
              • Check length and value range
              • Check for compliance with the format requirements of the target mission
            • Use Cases
              • Normative inspection of formats
              • Issues that can be judged by clear rules
            • Limitations
              • Can’t judge if the complex semantics are correct.
          • Functional Correctness Test
            • Principles
              • Performing the operation or result in the sample and judging its correctness by actual behaviour
            • Practical Methods
              • Code Generation
                • Compile Code
                • Run Cell Test
                • Check the test results.
              • Tool Call
                • Check tool names and parameters
                • Implementation tool call in isolation
                • Check tool implementation results
              • Math questions.
                • Validate results using a certainty calculation or symbol tool
            • Use Cases
              • Existence of clear, enforceable standards of correctness
            • Characteristics
              • Validation results are relatively objective
              • Certification costs depend on the implementation environment
          • Fact Checking
            • Principles
              • Comparison of factual statements in the sample with credible evidence
            • Practical Methods
              • Factuals for validation of extraction
              • Find credible references
              • Consistency between comparative presentations and references
              • Mark correct, wrong or unverifiable content
            • Use Cases
              • Knowledge questions and answers
              • Factual content generation
            • Limitations
              • There may be errors in the reference itself.
              • Can’t confirm it’s not necessarily a mistake.
          • Decision AI (LLM-as-a-Judge)
            • Principles
              • Use of models to assess data quality against predefined standards
            • Practical Methods
              • Definition assessment criteria (Rubric)
                • Correctness
                • Relevance
                • Mission-compliant
                • Coherence, etc.
              • Enter a sample to be evaluated
              • Require model output rating or satisfactory judgement
              • Screening low-quality samples based on thresholds
            • Use Cases
              • Open Response
              • Data that are difficult to verify using certainty rules
            • Limitations
              • Possible assessment deviations and miscalculation
              • Need manual sampling calibration
          • Manual review
            • Principles
              • Manual determination of whether the sample meets predefined quality standards
            • Practical Methods
              • Random sample
              • Scattered by task category
              • Focus on reviewing low-scoring and controversial samples
              • Checks if the label matches the guidelines
              • Multimarker Crosscheck
            • Use Cases
              • Complex semantic judgment
              • High-risk or automated data difficult to verify
            • Limitations
              • Higher cost
              • There may be a subjective bias.
        • Metrics
          • Sample eligibility rate
            • Number of samples examined / Total number of samples examined
          • Label Correct Rate
            • Number of correctly marked samples / Total number of verified samples
          • Note Consistency
            • Level of consistency in the judgement of the same sample by the various labelers
          • Format pass rate
            • Number of samples meeting format requirements / Total number of samples examined
        1. Data Coverage Validation
        • What It Is
          • Checks whether the data set adequately covers the target mission and its use scene
        • Why It Is Needed
          • The high quality of the data does not mean adequate coverage
          • Samples may be concentrated on a few common tasks.
          • Missing scenes may lead to underperformance of models in practical applications
        • Implementation Methods
          • scene coverage check
            • Principles
              • Comparison of data set coverage with predefined target scenarios
            • Practical Methods
              • Create a scene list based on training targets
              • Map the sample to the corresponding scene
              • Statistics covered and not covered
            • Metrics
              • Area coverage
                • Number of target scenes/ total target scenes covered
          • Sample distribution analysis
            • Principles
              • Statistics of the percentage of samples in different tasks, fields and forms of input output
            • Practical Methods
              • By task type
              • By thematic area
              • Group by input and output
              • Analysis of sample distribution of different scenarios
            • Metrics
              • Number and percentage of samples per scene
              • Number of active samples from key scenes
          • Missing scene recognition
            • Principles
              • Compare the target scene with the current data distribution, find the under-covered
            • Practical Methods
              • Recognize completely missing scenes
              • Identification of scenes with insufficient number of samples
              • Identification of scenes lacking anomalies and border conditions
              • Data replenishment plan based on gaps
        • Notes
          • The high coverage doesn’t mean there’s enough sample for each scene.
          • There’s no need to pursue diversity that has nothing to do with the mission.
          • Key scenes can set higher coverage requirements
        1. Data Quantity Validation
        • What It Is
          • Check whether the effective data scale meets the intended training requirements
        • Why It Is Needed
          • The number of original samples does not equal the number of valid samples
          • Weighting and filtration may reduce the size of the data significantly.
          • Different data requirements for different missions and training methods
        • Implementation Methods
          • Valid sample statistics
            • Principles
              • Statistical samples that can still be used for training after quality check, weighting and filtering
            • Practical Methods
              • Statistics of original samples
              • Number of invalid or duplicate samples
              • Statistics of final effective samples
            • Metrics
              • Number of active samples
              • Data retention rate
          • Data scale requires inspection
            • Principles
              • Comparison of actual and effective sample sizes with target size for data development phase
            • Practical Methods
              • Number of total samples checked
              • Samples of key scenes checked
              • Check if the training, validation and testing sets have sufficient data
            • Notes
              • Number of fixed samples not applied to all models and missions
        1. Data compliance and pollution inspection
        • What It Is
          • Check whether data use is legally compliant and whether data leaks have the credibility of impact assessments
        • Why It Is Needed
          • Training data may contain sensitive information or restricted content
          • Duplication of training and assessment sets may lead to a low level of assessment results
          • Possible non-transparent sources of synthetic data
        • Implementation Methods
          • Privacy and compliance inspections
            • Principles
              • Check the conditions for use of data in accordance with laws, permits and internal policies
            • Practical Methods
              • PII Test
                • Testing of personal sensitive information using rules or entity identification models
              • Review of data sources
                • Check open data, license and use of proprietary data
              • Synthetic Data Trace
                • Record generation models, source of raw data and process of generation
            • Results of acceptance and inspection
              • Identification of non-compliant samples
              • Decision to delete, sensitize or restrict use
          • Data Pollution
            • Principles
              • Check whether training data overlap with independent evaluation data
            • Practical Methods
              • Exact Repeat Check
                • Identification of the same samples by, inter alia, Hash
              • Approximate double check
                • Identification of highly overlapping content through MinHash, text similarities, etc.
              • Semantic Overlay Check
                • Check a sample of different expressions but with highly overlapping answers according to mission
              • Synthetic Data Trace
                • Analysis of the possibility that synthesis processes may introduce assessment data
            • Notes
              • There’s no guarantee that there’s no data contamination.
              • The assessment set should be as independent as possible from the training data build process
      • Receiving and inspection and feedback
        • Summarize validation results
          • Quality indicators
          • Coverage indicator
          • Number of valid data
          • Compliance and pollution inspection results
        • Determination of whether or not the predefined criteria have been met
          • Satisfactory
            • Output of accepted data set
          • Failure to meet requirements
            • Quality Issues
              • Return data processing or data labelling phase
            • Undercovered
              • Returns data acquisition and synthesis phase
            • Inadequate number
              • Additional valid sample
            • Compliance or pollution issues
              • Segregate, delete or reconstruct data
      • Downstream feedback
        • Use of DSD training model
        • Assess model performance on independent assessment sets
        • Adjustment of data requirements and processing strategy to training effectiveness
      • Notes
        • Data validation should go through data acquisition, synthesis and processing
        • The final acceptance doesn’t mean that data will improve model performance.
        • Training impact assessment is downstream feedback to validate the actual value of data

Discussion

Sign in with GitHub to comment. Discussions are stored as GitHub Issues.View on GitHub