Our recipe for rebuilding all the scihist_digicoll servers from scratch.
1. Build
There’s no need for downtime during this phase, as building new servers should in no way interfere with the existing ones.
If there’s no rush, consider performing the build and switch phases on consecutive days.
DB
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=database tier=production"
Add DB’s private IP to group_vars/kithe_production
redis_ip: new_private_ip_of_scihist_digicoll-database1-production
postgres_ip: new_private_ip_of_scihist_digicoll-database1-production
SOLR
ansible-playbook create_kithe.yml --vault-password-file password_file.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=solr tier=production"
Add SOLR’s private IP to group_vars/kithe_production
solr_ip: new_private_ip_of_scihist_digicoll-solr1-production
Jobs
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=jobs tier=production"
Web
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=web tier=production" --extra-vars "@group_vars/group_vars/kithe_web_production_override"
Deploy
Deploy master branch to all servers.
At this stage, if you point the prod URL in your etc/hosts
file locally to the web server, you should see an empty production website.
2. Switch
Point the prod URL in your
etc/hosts
file locally to the web server.Spin up the downtime server
Point the elastic IP for the prod web server to the downtime server
Spin down the existing prod servers
Copy the old database over to the new DB server.
Index
bundle exec cap production invoke:rake TASK="scihist:solr:reindex"
Point elastic IP to the new web server:
Select "Elastic IPs"
Select "digicoll-production"
Actions -> Associate address
Under the "Resource type", Choose "Instance".
Fill in the appropriate instance (the prod web server)
Check "Allow this Elastic IP address to be reassociated if already attached"
Click "Associate".
Remove the
etc/hosts
entry and check that traffic is indeed routed to the new serverAnnounce the switch is complete
3. Wrap up
Wait a couple days
Delete the old servers in AWS console
Delete the old disks left orphaned by the old servers