AI Engineering Architecture and User Feedback

1. What It Is

AI Engineering Architecture
- What It Is
  - Combining basic models with context-building, safety protection, route, cache, Agent and observability to form a complete AI application system
- Core Principle
  - Progressive architecture evolution
    - Start with simple Model API call
    - Progressive addition of components based on actual problems
    - Avoiding premature introduction of unnecessary complexity
User Feedback
- What It Is
  - Collect user evaluation, behaviour and interactive information on AI applications to assess application performance and guidance system optimization
- Primary Uses
  - Application assessment
    - Judge whether the system meets user needs
  - Product optimization
    - Identification of practical problems and direction for improvement
  - Personalization
    - Adjust application performance to user preferences
  - Model optimization
    - Data for follow-up model training
Core Principle
- Evolve the Architecture Incrementally and Improve It Through Feedback
- Evaluate and monitor the application → collect feedback → identify problems → improve the system → evaluate again

2. Why It Is Needed

A Single Model API Call Is Not Enough for Production Systems
- Missing External Knowledge
  - Needs RAG and Context Construction
- Incorrect or Unsafe Outputs
  - Need Guardrails
- Different Tasks Benefit from Different Models
  - Need Router
- Managing Multiple Model APIs Is Complex
  - Model Gateway required
- Repeated Requests Increase Latency and Cost
  - Cache.
- Complex Tasks Require Dynamic Planning and Execution
  - Agent required
- Cross-Component Execution Is Hard to Orchestrate
  - Need Orchestrator
- Failures Across Components Are Hard to Diagnose
  - Need Observability
AI Systems Introduce Additional Uncertainty
- The Same Input Can Yield Different Outputs
- Outputs May Violate Required Formats
- Models Can Hallucinate
- Model Behavior Can Change Between Versions
- Prompts Can Be Targeted by Injection Attacks
- Multi-Step Agents Can Fail Partway Through Execution
Offline Evaluation Cannot Replace Real User Feedback
- Test Data Cannot Cover Every Real-World Scenario
- Better Technical Metrics Do Not Always Mean Happier Users
- Users May Not Express Their Needs Explicitly
- Real Usage Provides Evidence of Product Value

3. How to Implement It

Overall Workflow
- Evolve the Application Architecture
  - Base Model API
  - Enhance Context
  - Add Guardrails
  - Add Router and Model Gateway
  - Add Cache
  - Add
  - Introduction of Orchestra according to complexity
  - Observability goes through the process.
- Close the Feedback Loop
  - User use AI application
  - Collect user feedback and interactive data
  - Analysis of user satisfaction and failures
  - Identification of improvements
  - Optimize Prompt, Context, Model or system components
  - Implementation evaluation
  - Verify improvements
  - Release and continue to observe.
- Notes
  - Structure Evolution does not represent a fixed development sequence
  - Determination of the order of introduction of components according to actual needs
  - Observation should be considered from the design stage
Core Components
- Model API
  - What It Is
    - Provide basic model inference capabilities
  - Implementation Approaches
    - Third-party business model API
    - Custom hosting model services
- Context Construction
  - What It Is
    - Context required to complete the current task for modelling
  - Responsibilities
    - Retrieve External Information
    - Filter and organise data
    - Build Model Input
- Guardrails
  - What It Is
    - Detection and control of input-output risk
  - Responsibilities
    - Enter security check
    - Output quality and safety check
    - Failure Handling
- Router
  - What It Is
    - Select a process path or model according to the task
- Model Gateway
  - What It Is
    - Integrated access to and management of multiple model services
- Cache
  - What It Is
    - Cache Reusable Request or Execution
- Agent and Tools
  - What It Is
    - Support dynamic decision-making, tool mobilization and environmental interaction
- Orchestrator
  - What It Is
    - Coordination of implementation processes and data transmission for multiple components
- Observability
  - What It Is
    - Records and analysis system operational to help identify and locate problems
  - Cross-cutting capability
    - Through the system.
    - Not necessarily independent.
- Feedback System
  - What It Is
    - Collection and analysis of user feedback
  - Responsibilities
    - Feedback collection
    - Feedback analysis
    - Provide a basis for improvement
- Notes
  - This is the logical division of responsibilities.
  - No requirement for each component to be deployed independently
  - Multiple functions can be performed by the same component
Implementation principles and technology options
- 1. Context Construction
  - What It Is
    - Prepare relevant information for the model for the current task
  - Implementation Approaches
    - RAG
      - Retrieving information from external sources of knowledge
    - Tool Calling
      - Get real time information or execute operations
    - File Upload
      - Provide external documentation to models
    - Other data retrieval
      - Database
      - Table
      - Image
  - Relationship to traditional engineering
    - Like Feature Engineering
      - Prepare effective input for models
  - Detailed principles
    - Repeat Chapter 6 RAG and Agent Knowledge
- Guardrails (security)
  - What It Is
    - Detection and treatment of risks and failures in model input and output
  - Classification
    - Input Guardrails (input protection)
      - Problem Addressed
        
        Sensitive data leaks
        
        Malicious hints
        
        Prompt Injection
      - Implementation Methods
        
        Testing of Personal Identification (PII)
        
        Testing of business sensitive information
        
        Intercepting high-risk inputs
        
        Sensitive information mask
        
        Restore raw information through a security map if necessary
      - Implementation process
        
        Enter Test
        
        Identification of sensitive information
        
        Interception or dissensitivity
        
        Send securely processed input
    - Output Guardrails (output protection)
      - Problem Addressed
        
        Model output may not meet quality or safety requirements
      - Type of failure
        
        Format error
        
        JSON doesn’t match Schema.
        
        Factual error
        
        Generate non-existent or inaccurate information
        
        Quality Issues
        
        Answers are irrelevant or of poor quality
        
        Harmful content
        
        Violation of applicable security requirements
        
        Privacy disclosure
        
        Output Sensitive Information
        
        Dangerous movement.
        
        Trigger unauthorized external operations
      - Test Method
        
        Procedural rules
        
        Specialized test model
        
        AI Scorer
      - Failure Handling
        
        Try again
        
        Refuse Response
        
        Use of backup schemes
        
        Transfer Manual
  - Main trade-offs
    - Security and delay
    - Safety and costs
    - Stream output and full content check
  - Notes
    - Safety protection does not completely eliminate risk.
    - Need to focus on the rejection rate.
    - Retesting increases costs and delays
    - High-risk action also requires executive access control.
- Router
  - What It Is
    - Select the path for processing according to task type, context and execution requirements
  - Implementation Approaches
    - Intent Classification
      - What It Is
        
        Identification of user intentions and selection of corresponding treatment options
      - Example
        
        Get the password back.
        
        Billing dispute.
        
        • Technical support for Agent
      - Other uses
        
        Identification of requests beyond service
        
        Identification of vague requests
        
        Select the model by task complexity
    - Next-action Predation
      - What It Is
        
        Forecast the next action to be implemented based on current status
      - Example
        
        Select which tool to call
        
        Judge whether or not to search
        
        Select the corresponding memory storage
  - Achieving technology
    - Rule route
    - Classification Model
    - Small Language Model
  - Optimizing objectives
    - Reduce unnecessary high-cost model calls
    - Reduce the delay and cost of route by itself
- Model Gateway (model gateway)
  - What It Is
    - Providing a unified, secure, managed access interface for different model services
  - Main functions
    - Unified model API
      - Adapted different model providers
    - Access control
      - Manage model privileges for users and applications
    - Cost management
      - Record and limit API calls
    - Rate Limiting
      - Limit the frequency of requests
    - Failover
      - Toggle backup if the main model is not available
    - Load Balancing
      - Distribution model requests
    - Logging
      - Record model call information
    - Optional Functions
      - Cache
      - Safety
  - Relationship to traditional engineering
    - API Gateway
    - Adapter Pattern
  - Robert and Gateway.
    - Router
      - Decision on which model or treatment to use
    - Gateway
      - Harmonization of calls and management model services
    - The two can be combined.
- Cache (Cache)
  - What It Is
    - Reuse existing calculations to reduce the cost and delay of duplicate requests
  - Classification
    - Exact Cache
      - What It Is
        
        Use the result directly when matching the cache key
      - Implementation Approaches
        
        Find results using cache keys
        
        Hit directly back
        
        Miss Execute Model Call and Cache
      - Storage Scheme
        
        Memory
        
        Redis
        
        PostgreSQL
      - Phase-out strategy
        
        LRU
        
        LFU
        
        FIFO
    - Semantic Cache
      - What It Is
        
        Allows a similar semantic request to repeat the cache result
      - Implementation Principles
        
        Embedding
        
        Convert Query to Vector
        
        Vector Search
        
        Retrieving similar queries
        
        Similarity Threshold
        
        Assess whether similarity exceeds the threshold
      - Execution Flow
        
        Query Query Embeding Vector Search Similarity
        
        Return Cache Result by Hit
        
        Call models and update caches if not hit
  - The difference between the two caches
    - Exact Cache
      - Make it simple.
      - It’s a narrow hit range.
      - The risk of error reuse is relatively easy to control
    - Semantic Cache
      - The range of hits is wide.
      - Complicated
      - Could be wrong to match different problems
      - Additional vector search costs required
  - Cache risk
    - Data timeliness
    - User Private Disclosure
    - Over-authorization error reuse
    - Semantic matching error
  - Notes
    - Consider Exact Cache.
    - Semantic Cache needs to verify the actual proceeds.
    - Cache keys and hit conditions should take into account privileges and business context
- 1. Agency Pattersons (intelligence model)
  - What It Is
    - Support dynamic decision-making, recycle implementation, conditionality branches and tools
  - Why It Is Needed
    - Fixed linear processes cannot meet some of the complex tasks
  - Implementation Approaches
    - ReAct
      - The reasoning, action, observation, observation, continuation of the reasoning.
    - Tool Calling
      - Retrieve External Information
      - Execute External Operations
    - Complex implementation model
      - Loop
      - Conditional Branch
      - Parallel Execution
  - Writer Actions
    - What It Is
      - Operation to modify the external environment
    - Example
      - Send Mail
      - Order.
      - Update database
    - Risk
      - It could have real external side effects.
      - Competency control and necessary clearance required
  - Details
    - Re-enact Chapter 6Agent and Harness Knowledge
- 1. Observability
  - What It Is
    - Analysis of system locations and operational data, understanding of system status and positioning failures
  - Why It Is Needed
    - Reduced application failure and security risk
    - Identification of performance and cost optimization opportunities
    - Helping to locate causes of failure of complex components
  - Core technology
    - Metrics (indicator)
      - What It Is
        
        Measurement of system status and performance through numerical measurements
      - Classification of indicators
        
        Model quality
        
        Mission Right Rate
        
        Factual Consistency
        
        Format Error Rate
        
        Safety and Security
        
        Percentage of harmful content
        
        Rate of rejection
        
        Error rejection rate
        
        Performance
        
        TTFT
        
        TPOT
        
        Total Response Time
        
        Cost
        
        Token Usage
        
        Number of requests
        
        API cost
        
        Cache Hit Rate
        
        Component Performance
        
        Retrieval delay
        
        Search quality
        
        Vector database query time-consuming
        
        User Behaviour
        
        Dialogue rounds
        
        Regeneration Rate
        
        Termination rate
        
        Operational indicators
        
        DAU
        
        Subscriptions
        
        Conversion Rate
    - Logging
      - What It Is
        
        Record specific events that occurred during the operation of the system
      - Record contents
        
        Model & Call Configuration
        
        Prompt and its version
        
        Query and Response
        
        Tool call and implementation results
        
        Error message
        
        Component execution time
        
        Request Link ID
      - Role
        
        Rewind specific issues
        
        Analyse anomalies
    - Tracing
      - What It Is
        
        Associated with the events of the various components in the execution of the request and re-establishing the complete implementation path
      - Record contents
        
        Component Call Order
        
        Intermediate results
        
        Time-consuming execution
        
        Token Usage
        
        Call Costs
        
        Instructions
      - Role
        
        Which component or step to locate the failure
    - Drift Design
      - What It Is
        
        Detection of changes in system input, configuration or model behaviour over time
      - Common type
        
        Prompt Drift
        
        Changes in hint templates
        
        User Behavior Drift
        
        Changes in user search habits
        
        Model Drift
        
        Changes in bottom model behaviour
  - Traditional DevOps indicators
    - MTTD
      - Average test time
      - From problem to discovery
    - MTTR
      - This book means average response time
      - From testing problems to solving problems
    - CFR
      - Change failure rate
      - Rate of change causing failure or needing repair, rollback
  - Core principles
    - Let’s determine the pattern.
    - Design indicators and alarms.
    - A needs-based combination of sampling and comprehensive inspection
    - Remittal of surveillance to offline assessment of currents
- 1. Orchestration
  - What It Is
    - Coordinate implementation and data flow of multiple models, tools and data sources
  - Implementation Principles
    - Component Defense
      - Model used in the declaration system
      - Declaration of data sources and tools
      - Statement assessment and monitoring component
    - Chaining
      - Define component execution sequence and data dependency
  - Key competencies
    - Data transmission
    - Format Conversion
    - Conditional Branch
    - Parallel Execution
    - Abnormal treatment
  - Analogies in Traditional Engineering
    - Workflow Engine
    - Pipeline
  - Common tools
    - LangChain
    - LlamaIndex
    - Haystack
    - Flowise
    - Langflow
  - Selection criteria
    - Integration and scalability
    - Complex flow line support
    - Usability
    - Performance
    - Scalability
  - Author’s Proposal
    - Initial priority attempt not to rely on a complex framework
    - When the complexity actually increases, then we think of introducing it.
- 1. User Feedback (user feedback)
  - What It Is
    - Access to improved signals through user evaluation and interactive behaviour
  - Classification
    - Explicit Feedback
      - What It Is
        
        User initiative to provide clear evaluations
      - Example
        
        A little. A little.
        
        Star rating
        
        Manual evaluation
        
        The answer is better.
      - Characteristics
        
        Easy to explain.
        
        Additional user action required
        
        The number of feedbacks may be low
    - Implicit Feedback
      - What It Is
        
        Infer satisfaction and preferences through user behaviour
      - Natural Language Feedback (natural language feedback)
        
        What It Is
        
        Extract feedback from user dialogue content
        
        Signal
        
        Early termination
        
        User Stop Generation
        
        May indicate that the answer did not meet expectations
        
        Error fixes
        
        Other Organiser
        
        Could mean the model understands or answers wrong
        
        Restatement
        
        Other Organiser
        
        May indicate that the original issue has not been resolved
        
        Verification required
        
        User requests to re-examine or provide evidence
        
        Could indicate a lack of trust on the part of the user
        
        Direct Edit
        
        User modify AI generation
        
        Could create preference data
        
        He’s complaining.
        
        User indicates long, wrong or quality answers Bad
        
        Emotional change.
        
        Users show dissatisfaction or satisfaction
      - Other Interaction Feedback (other interactive feedback)
        
        Regenerated
        
        Possible dissatisfaction
        
        Maybe it’s just a comparison.
        
        Delete Dialogue
        
        Possible unsatisfactory dialogue
        
        Maybe for privacy reasons.
        
        Collection and Sharing
        
        Could reflect user preferences
        
        Length of dialogue
        
        Different meanings in different operations
        
        Diversity of dialogue
        
        Auxiliary Identification Invalid Repeat Dialogue
      - Characteristics
        
        Data sources are abundant
        
        No user-specific rating required
        
        More noise and ambiguity.
  - Feedback Design
    - When do you collect it?
      - Initial phase of use
        
        Understand user preferences
        
        Necessary system calibration
      - There’s a problem.
        
        Collect bug information
        
        Allow users to correct
        
        Provision of manual assistance as necessary
      - When the model is uncertain
        
        Request user confirmation
        
        Comparison of candidate results
      - When a user gives a positive feedback
        
        Allows praises, collections, reviews, etc.
    - How to collect
      - Integration of feedback into normal work processes
      - Reduced cost of feedback operations
      - Allow Ignore Feedback Requests
      - Provide clear feedback options
      - Description of the use of feedback data
      - Control the frequency of feedback requests
      - Respect for privacy and authorization of users
    - Typical cases
      - GitHub Copilot
        
        Recommendation for code acceptance
        
        Ignore code suggestion
        
        Suggested changes
      - Midjourney
        
        Select Picture Zoom In
        
        Generate image variants
        
        Recreate Pictures
  - Feedback Use
    - Assessment of application performance
    - Guidance product improvements
    - User Personalization
    - Build preference training data
    - Generate Data Flying Wheel
Detailed Execution Flow
- 1. AI Request Implementation Process
  - User SendQuery
  - Input Guardrails
    - Check input security
    - Dissensitization or interception if necessary
  - Router
    - Other Organiser
    - Select Process Path
  - Cache Lookup
    - Check for safe reuse.
  - Context Construction (as required)
    - Retrieving external information
    - Filter & Organizational Context
  - Model Gateway
    - Call Target Model
  - LLM Generation
    - Generate model responses
  - Output Guardrails
    - Check output quality and safety
    - Retest or transfer manual as necessary
  - Return to Response
  - Notes
    - The order of implementation depends on the actual structure
    - Some steps can be omitted or parallel
    - Agent missions may include multi-wheeling tools
    - Observability goes through the whole process.
- 1. User feedback closed loop
  - User Received Response
  - Continue user interaction
  - Feedback Collection
    - Collect visible feedback
    - Extract hidden feedback
    - Collect the necessary interactive context
  - Feedback Analysis
    - Other Organiser
    - Analysis of satisfaction
    - Analyse preferences and feedback deviations
  - Setting directions for improvement
    - Prompt Optimisation
    - Context Optimization
    - Model optimization
    - System component optimization
  - Perform offline assessment
  - Verify improvements
  - Issued and observed on an ongoing basis

4. How to Optimize It

Optimizing system quality
- Inadequate knowledge
  - RAG and Context Construction
- Model selection is not reasonable
  - Router
- Output format or quality issues
  - Output Guardrails
- Complex mission failure
  - Agentic Patterns
- Complex workflows difficult to maintain
  - Orchestrator
Optimization of delays and costs
- Cache Optimization
  - Priority Exact Cache
  - Introduction of Semantic Cache as required
- Model route
  - Simple missions use low-cost models
  - Use of more capable models for complex missions
- Parallel Execution
  - Parallel treatment of separate steps
- Failure Handling
  - Use retest or backup models
  - Balancing the cost and delay of the re-test
Optimizing reliability
- Input Output Validation
- Model failure back.
- Unusual retry
- Manual intervention
- Key Operating Permission Controls
Optimization of detectability
- Clear failure pattern.
- Design Metrics around Fault
- Logs Logs and Traces
- Set up the alarm.
- Positioning issues
- Add failure cases to regression assessment
Optimizing the quality of user feedback
- Main deviation
  - Lenity Bias
    - What It Is
      - Users tend to rate more than real experience
    - Reasons
      - Conflict avoidance
      - Negative evaluations require additional operations
  - Randomness (random)
    - What It Is
      - Select feedback randomly without careful comparison
  - Profile Bias
    - What It Is
      - Options display location influence user selection
    - Mitigation
      - Randomly reposition options
      - Statistical correction by location
  - Preference Bias
    - What It Is
      - Certain sensory features affect user selection
    - Example
      - User preference for longer answers
      - But a longer answer may not be more accurate.
  - Recency Bias
    - What It Is
      - The answer to what the user might have preferred
- Optimization principle
  - Reduce unnecessary feedback operations
  - Cross-check multiple signals
  - Check for sample distribution of feedback
  - Avoid directly equating user preferences to correctness
Degenerate Feedback Loop
- What It Is
  - The system output affects user feedback, while the feedback affects the next round of system outputs, resulting in a continuous widening of the initial deviation
- Implementation mechanisms
  - Something is given priority.
  - More exposure and interaction
  - The system thinks it’s more popular.
  - Continue to raise its priorities
  - Form a self-reinforcing cycle
- Possible problems
  - Exposure bias
  - Pop differential
  - Information Cocoon Room
  - Data distribution is becoming more and more homogeneous.
  - The original bias of the model is magnified.
- Sycophancy.
  - What It Is
    - Models tend to accommodate user views in order to obtain user approval
  - Risk
    - User preferences raised
    - But the truth may fall.
- Core principles
  - User feedback should not be the only target for optimization
  - Need to combine objective correctness and operational constraints

5. How to Evaluate It

Evaluation thinking
- Reuse chapter 4 assessment methodology
- Extension from model-level assessment to entire application systems
- Offline assessment combined with online monitoring
Assessment dimensions
- Functional correctness
  - Mission success rate
  - Format Correct Rate
- Generate Mass
  - Correct answer.
  - Factual Consistency
  - Relevance
- Safety and Security
  - Risk detection rate
  - Error rejection rate
- Reliability
  - Request failure rate
  - Retry Rate
- Performance
  - TTFT
  - TPOT
  - Total Response Time
- Cost
  - Token Usage
  - Single task cost
- User Experience
  - User satisfaction
  - Regeneration Rate
  - User correction rate
- Observability
  - MTTD
  - MTTR
- Business value
  - DAU
  - Conversion Rate
  - Subscriptions
Modalities of assessment
- Offline assessment
  - What It Is
    - Testing the system using pre-prepared data
  - Use Cases
    - Develop Authentication
    - Version returns
    - Programme comparison
  - What to Evaluate
    - Model quality
    - RAG Retrieving Quality
    - Agent mission success rate
    - Output Format
    - Security constraints
- Online assessment and monitoring
  - What It Is
    - Use of data observation systems in real production environments
  - Data sources
    - Metrics
    - Logs
    - Traces
    - User Feedback
    - Use AI scorer if necessary
  - Main objectives
    - Production environment failure detected
    - Monitoring model quality
    - Recognize input and changes in user behaviour
    - Identify opportunities for optimization
- Notes
  - Observability provides operational data and problem positioning capabilities
  - Normal indicators don’t make the mission a success.
  - The right answer still requires a proper scoring mechanism
Assessment and optimization of closed loops
- Collection of assessment data
- Implementation evaluation
- Analysis of corporate indicators
- Analyse failure cases
- Positioning issues
- Proposals for improvements
- Implementation improvements
- Re-evaluate.
- Add evaluation data for failure cases Set