Our recipe for rebuilding all the scihist_digicoll servers from scratch.

1. Build

DB server
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=database tier=production"

Add DB’s private IP to group_vars/kithe_production

redis_ip: new_private_ip_of_scihist_digicoll-database1-staging
postgres_ip: new_private_ip_of_scihist_digicoll-database1-staging

SOLR server
ansible-playbook create_kithe.yml --vault-password-file password_file.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=solr tier=production"

Add SOLR’s private IP to group_vars/kithe_production

solr_ip: new_private_ip_of_scihist_digicoll-solr1-staging

Jobs
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=jobs tier=production"

Web
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=web tier=production" --extra-vars "@group_vars/group_vars/kithe_web_production_override"

At this stage, if you point the prod URL in your etc/hosts file locally to the web server, you should see an empty production website.

2. Switch

• Point the prod URL in your etc/hosts file locally to the web server.

• Spin up the downtime server

• Point the elastic IP for the prod web server to the downtime server

• Spin down the existing prod servers

• Deploy master branch to all servers

• Copy the old database over to the new DB server.

• Index
  bundle exec cap staging invoke:rake TASK="scihist:solr:reindex"

• Point elastic IP to the new web server:
  Select "Elastic IPs"
Select "digicoll-production"
Actions -> Associate address
Under the "Resource type", Choose "Instance".
Fill in the appropriate instance (the prod web server)
Check "Allow this Elastic IP address to be reassociated if already attached"
Click "Associate".

3. Wrap up

• Wait a couple days

• Delete the old servers in AWS console

• Delete the old disks left orphaned by the old servers