Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Our recipe for rebuilding all the scihist_digicoll servers from scratch.

1. Build

DB
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=database tier=production"

Add DB’s private IP to group_vars/kithe_production

redis_ip: new_private_ip_of_scihist_digicoll-database1-production
postgres_ip: new_private_ip_of_scihist_digicoll-database1-production

SOLR
ansible-playbook create_kithe.yml --vault-password-file password_file.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=solr tier=production"

Add SOLR’s private IP to group_vars/kithe_production

solr_ip: new_private_ip_of_scihist_digicoll-solr1-production

Jobs
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=jobs tier=production"

Web
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=web tier=production" --extra-vars "@group_vars/group_vars/kithe_web_production_override"

Deploy

Deploy master branch to all servers.

(lightbulb) At this stage, if you point the prod URL in your etc/hosts file locally to the web server, you should see an empty production website.

2. Switch

  • Point the prod URL in your etc/hosts file locally to the web server.

  • Spin up the downtime server

  • Point the elastic IP for the prod web server to the downtime server

  • Spin down the existing prod servers

  • Copy the old database over to the new DB server.

  • Index
    bundle exec cap production invoke:rake TASK="scihist:solr:reindex"

  • Point elastic IP to the new web server:

    • Select "Elastic IPs"

    • Select "digicoll-production"

    • Actions -> Associate address

    • Under the "Resource type", Choose "Instance".

    • Fill in the appropriate instance (the prod web server)

    • Check "Allow this Elastic IP address to be reassociated if already attached"

    • Click "Associate".

  • Remove the etc/hosts entry and check that traffic is indeed routed to the new server

  • Announce the switch is complete

3. Wrap up

  • Wait a couple days (question)

  • Delete the old servers in AWS console

  • Delete the old disks left orphaned by the old servers

  • No labels