1.13 Distributed-System Upgrade and Rollback

1. What Are Service Upgrade and Rollback?

• A service in a distributed system has multiple instances. When a new feature is released or an old system is refactored, all of these instances need to be upgraded and replaced. If a problem occurs, rollback needs to happen promptly.

2. Deployment Strategies

2.1. Downtime Deployment

• Stop the existing version of the service and then deploy the new version.
• Advantage: the old and new versions will not be online at the same time during deployment, ensuring consistency.
• Disadvantage: downtime is required and has a large impact on users.

2.2. Blue-Green Deployment

• Prepare two environments. Traffic goes to environment A. After publishing the new version to environment B, switch traffic to environment B.
• Advantage: no downtime is required, and there is no consistency problem caused by old and new versions being online at the same time.
• Disadvantage: two environments are required, wasting resources; it is not very friendly to stateful services.

2.3. Rolling Deployment

• Replace all instances of the application one by one to gradually release a new version of the application.
• Advantage: friendly to stateful services.
• Disadvantages: consistency problems caused by old and new versions being online at the same time; released directly without verification.

2.4. Canary Deployment

• Switch some users to the new version and check whether there are any problems. If there are no problems, continue expanding the upgrade until all users have been upgraded.
• Release process 2. Adjust machine weight to 0. 3. Release. 4. Adjust machine weight to 10. 5. Log in to the machine and inspect logs. 6. After confirming there are no problems, adjust machine weight to 100. 7. Continue with the other machines.
• Rollback process
  1. Roll back (that is, republish using the old package).
  2. Keep one machine’s weight at 0.
  3. Add logs and change the log level in the configuration file to debug.
  4. Publish the package to that machine.
  5. Send a request to that machine by specifying its IP, then log in to the machine and inspect the logs.

2.5. A/B Testing

• Put two versions online at the same time and make relevant comparisons.
• Canary deployment vs A/B testing
  • Canary deployment: lack of confidence in the quality of the new version.
  • A/B testing: lack of confidence in the functionality of the new version.

3. Rollback Strategies

3.1. Installation-Package Rollback

• Redeploy the old installation package.

3.2. Switch Rollback