Upgrade Steps
Switch the settings for Ruby in the grou_vars file(s) you want to use
Use the build_ami script to make a new ruby AMI.
Update the group_vars files to use the new AMI.
Switch Box steps
- Leave current prod up
- Build new prod boxes for app, jobs, and solr.
- Once boxes are done, alert staff to stop all edits and ingests
- Make sure new prod boxes are fully connected by running update-box.yml on the new boxes.
- Cap deploy to new prod boxes
- Make sure cap gives no errors.
- Run a backup of solr
- Backup app's postgres DB and Redis
- On new production boxes
- In new-app drop the blank chf_hydra db, create a new blank chf_hydra db and then follow the postgres import instructions in Backups and Recovery
- Stop Redis, delete the existing redis.rdb in
var/lib/redis
- Copy the backed up redis.rdb over to the folder and make sure it is owned by redis
- Restart Redis
- Stop Redis, delete the existing redis.rdb in
- In new-solr run the solr restore commands in Application administration
- In new-app drop the blank chf_hydra db, create a new blank chf_hydra db and then follow the postgres import instructions in Backups and Recovery
- Edit New-Prod App to allow IP access without directing to actual prod by disabling apache redirect
- This is not completely needed, but is a way to be sure there were no problems
- Check that New-Prod works properly with solr's index being loaded
- Return New-Prod App to normal apache configuration
- Move Elastic IP from app-prod to the new app-prod.
- Make sure all details in ansible's group_vars are up to date with any internal IP address changes and commit and merge with master.
Revert Box steps
Assuming the old boxes were not deleted. Follow these steps.
- Start the old servers up.
- Double check their internal IP addresses, they should be the same as before but always be sure.
- Check netdata to be sure they have resque workers running
- Tell users to stop adding new items
- Run a backup of solr
- Backup app's postgres DB and Redis
- On new production boxes
- In new-app drop the blank chf_hydra db, create a new blank chf_hydra db and then follow the postgres import instructions in Backups and Recovery
- Stop Redis, delete the existing redis.rdb in
var/lib/redis
- Copy the backed up redis.rdb over to the folder and make sure it is owned by redis
- Restart Redis
- Stop Redis, delete the existing redis.rdb in
- In new-solr run the solr restore commands in Application administration
- In new-app drop the blank chf_hydra db, create a new blank chf_hydra db and then follow the postgres import instructions in Backups and Recovery
- Edit New-Prod App to allow IP access without directing to actual prod by disabling apache redirect
- This is not completely needed, but is a way to be sure there were no problems
- Check that New-Prod works properly with solr's index being loaded
- Return New-Prod App to normal apache configuration
- Move Elastic IP from app-prod to the new app-prod.
- Make sure all details in ansible's group_vars files are up to date with any internal IP address changes and commit and merge with master.