Scripts for production deployments

[cclauss]

Closes #4382
This is a temporary script-based Open Library production deployment that will be refactored into fabfile.py.
It allows us to make sure all steps are clear and complete before migrating them into fabfile.py.
https://github.com/internetarchive/openlibrary/wiki/Deployment-Scratchpad

• ol-home0: time start_production_deploy.sh
  1. ol-home0, ol-covers0, web{1, 2}: continue_production_deploy.sh
  2. User presses return when ready...
  3. ol-home0, ol-covers0, web1: finish_production_deploy.sh
  4. User presses return when ready...
  5. ol-web2: finish_production_deploy.sh

Technical

Testing

https://github.com/internetarchive/openlibrary/wiki/Deployment-Scratchpad

Screenshot

Stakeholders

[cclauss]

Roadmap... (This PR or future iterations)

• timestamp vs. SHA (or better yet, do BOTH)
  • docker image ls already tells us when each image was created so a timestamp is not required
• rsync the four repos from ol-home0
• booklending_utils is only needed on ol-web{1, 2}
• ssh the commands
• split continue_production_deploy.sh in two so docker-compose down/up happens just-in-time
• uncomment cronjobs (Create cron jobs docker-compose service #4383) importbot (Create importbot docker service #4324) solr-updater

[mekarpeles]

lgtm;

• questions about how volume-mount affects ability to rollback
• questions about having web restarts occur in deployment script itself (currently we deploy and then restart independently with the intent that a human operator is available to rollback after the first failure experienced during blue/green)

[mekarpeles]

@cclauss and I agreed that given this is the process we're currently using, we can go ahead and land this initial set of changes and open an issues for behavior change which needs to occur as a followup.